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FOREWORD 


1 .  This  standard  defines  the  Government’s  requirements  and  expectations  for  contractor 
performance  in  defense  system  acquisitions  and  technology  developments. 

2.  This  new-issue  SMC  standard  comprises  the  text  of  The  Aerospace  Corporation  report 
number  TOR-2007(8583)-6414,  Volume  1,  Rev.  A,  entitled  Technical  Reviews  and  Audits  for 
Systems,  Equipment,  and  Computer  Software. 

3.  Beneficial  comments  (recommendations,  changes,  additions,  deletions,  etc.)  and  any 
pertinent  data  that  may  be  of  use  in  improving  this  standard  should  be  forwarded  to  the  following 
addressee  using  the  Standardization  Document  Improvement  Proposal  appearing  at  the  end  of  this 
document  or  by  letter: 


Division  Chief,  SMC/EAE 
SPACE  AND  MISSILE  SYSTEMS  CENTER 
Air  Force  Space  Command 
483  N.  Aviation  Blvd. 

El  Segundo,  CA  90245 

4.  This  standard  has  been  approved  for  use  on  all  Space  and  Missile  Systems 
Center/Air  Force  Program  Executive  Office  -  Space  development,  acquisition,  and 
sustainment  contracts. 


DAVID  E.  SWANSON,  COL,  USAF 
SMC  Chief  Engineer 
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1.  Scope 

This  document  describes  the  objective  requirements  for  the  conduct  of  Technical  Reviews  and  Audits  on 
such  end  items  (Els)  as  a  system,  equipment,  distributed  and  embedded  Software  Configuration  Item 
(SWCI),  Hardware  Configuration  Item  (HWCI),  and/or  a  combination  thereof. 

This  document  is  an  update  to  MIL-STD-1521B  (USAF)  Technical  Reviews  and  Audits  for  Systems, 
Equipments  and  Computer  Software,  and  will  be  issued  as  a  USAF  Space  and  Missile  Systems  Center 
(SMC)  standard  (STD)  for  planning  and  execution  of  key  system  engineering  reviews.  This  update  also 
documents  a  desired  process  and  associated  criteria  for  technical  reviews  and  audits. 

This  document  is  composed  of  two  volumes: 

Volume  1  -  defines  a  generic  set  of  technical  reviews  and  audits  for  systems,  equipment,  and  computer 
software  Els  and  the  guidance  for  the  cost  effective  application  and  tailoring  of  this  standard  for  the 
reviews  described  in  Appendixes  A  through  K: 

Appendix  A  System  Requirements  Reviews  (SRR) 

Appendix  B  System  Functional  Reviews  (SFR) 

Appendix  C  Software  Requirements  and  Architecture  Review  (SAR) 

Appendix  D  Preliminary  Design  Reviews  (PDR) 

Appendix  E  Critical  Design  Reviews  (CDR) 

Appendix  F  Test  Readiness  Review  (TRR) 

Appendix  G  Functional  Configuration  Audit  (FCA) 

Appendix  H  Physical  Configuration  Audit  (PCA) 

Appendix  I  System  Verification  Review  (SVR) 

Appendix  J  Production  and  Manufacturing  Readiness  Review  (P/MRR) 

Appendix  K  Application  Guide  for  Tailoring  MIF-STD-1521 

Volume  2  -  provides  specific  and  unique  supplemental  criteria  content  for  the  core  technical  reviews 
(SRR,  SFR,  SAR,  PDR,  and  CDR)  in  Volume  1  for  Space  Systems  specific  equipment,  and  computer 
software  end  items  (Els)  to  be  implemented  per  the  contract  Statement  of  Work  (SOW).  These  criteria 
also  include  parts,  materials,  structures;  systems  engineering,  tests  and  processes. 

This  update  provides  the  Government’s  requirements  for  criteria-based  technical  reviews,  specified  under 
the  following  five  major  categories: 

1 .  Systems  engineering  and  architecture  development 

2.  System,  segment  and  subsystem  design 

3.  System  verification  and  validation 

4.  Engineering  disciplines  and  specialty  engineering 

5.  Integrated  technical  risk  and  mitigation 

of  the  end  item  (El)  development  maturity  and  operational  utilization  objectives  in  greater  detail  at  the 
following  core  reviews: 

Appendix  A  System  Requirements  Review  (SRR) 

Appendix  B  System  Functional  Review  (SFR) 

Appendix  C  Software  Requirements  and  Architectures  Review  (SAR) 

Appendix  D  Preliminary  Design  Review  (PDR) 

Appendix  E  Critical  Design  Review  (CDR) 
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The  update  also  identifies  and  delineates  criteria  for  selected  disciplines  and  specialty  areas  in  varying 
detail  and  the  associated  work  that  shall  be  performed  and  documented  in  support  of  the  above  core 
reviews  in  terms  of  generic  requirements  criteria  examples  across  all  DoD  systems  developments.  In 
addition,  this  update  also  defines  the  required  quality  attributes,  sufficiency,  and  progress  of  the 
contractor’s  documented  accomplishments,  along  with  the  engineering  disciplines  and  specialty 
processes,  to  be  presented  as  specific  evidence  of  the  contractor’s  accomplishments. 

Terms  and  definitions  related  to  general  technical  reviews  are  described  in  Section  3  of  this  document. 
The  general  technical  criteria  required  for  all  reviews  are  provided  in  Section  4.  Section  5  describes  the 
typical  working  relationships  and  roles  and  responsibilities  between  contractor  and  contracting  agents, 
which  encompass  process  steps  that  can  be  taken  in  preparation  for  and  closure  of  each  technical  review. 
The  content  of  Section  5  is  intended  as  source  material  for  planning  and  negotiations  for  conduct  of  the 
technical  reviews  between  the  contracting  agents. 

1.1  Purpose 

The  technical  reviews  and  audits  are  necessary  systems  engineering  (SE)  activities  performed  to  assess 
technical  progress  within  a  program,  relative  to  contractual  requirements  and  developmental  maturity. 

Technical  reviews  of  program  progress  shall  be  event-driven  and  conducted  when  the  system  under 
development  meets  the  review  entrance  criteria  as  documented  in  the  Systems  Engineering  Plan  (SEP). 
The  technical  reviews  and  audits  shall  include  participation  by  subject  matter  experts  who  are 
independent  of  the  program  (i.e.,  peer  review),  unless  specifically  waived  by  the  SEP  approval  authority 
as  documented  in  the  SEP. 

The  technical  reviews  are  conducted  by  the  contractors  and  subcontractors,  also  referred  to  as  primes  and 
subs  at  logical  transition  points  in  the  development  and  design  efforts,  for  an  idealized  El  in  various 
phases  of  an  El’s  development  to: 

1.  Determine  the  contractor’s  technical  progress  achieved  to  date 

2.  Compare  the  Els’  performance  against  the  requirements 

3.  Identify  potential  impediments  and  risks  to  each  El’s  design  execution 

4.  Determine  mitigation  plans  to  avert  program  schedule  delays  and  unplanned  resource 
expenditures. 

The  contracting  agency  shall  perform  initial  tailoring  of  this  document  in  accordance  with  (IAW) 
Appendix  K  Application  Guide  for  Tailoring  MIL-STD-1521,  to  require  only  what  is  needed  for  each 
individual  acquisition,  and  address  appropriate  program  scope,  program  size,  and  technical  progress 
within  the  acquisition  life  cycle. 

Technical  Reviews  and  Audits  defined  herein  shall  be  conducted  in  accordance  with  this  standard  to  the 
extent  specified  in  the  contract  clauses.  Statement  of  Work  (SOW),  Compliance  Standards  List,  and  the 
Contract  Data  Requirements  List  that  is  based  on  the  Government's  need  for  technical  data  required  to 
support  the  acquisition  and  life  cycle  support  strategies.  Guidance  in  applying  this  standard  is  provided  in 
Section  4  and  5. 

These  technical  reviews  provide  a  method  and  forum  for  the  primes  and  subs  and  the  contracting  agency 
to  assess  whether  the  status  of  the  end  item  under  development  and  the  supporting  documentation  have 
met  contract  requirements  and  expectations,  with  an  appropriate  level  of  maturity,  to  continue  to  the  next 
phase  of  the  program  with  manageable  risk. 

1.2  Objectives  of  Technical  Reviews 

DoDI  5000.02  Operation  of  the  Defense  Acquisition  System  (December  2008)  instruction,  require  that 
the  contracting  agent  schedule  and  conduct  technical  reviews  and  audits  (Table  1)  with  the  objectives  of 
enabling  means,  methods,  and  forum  that  provide  for  timely  and  critical  insight,  evaluation  and 
assessment  of  the  contractors  technical  progress  for  an  evolving  system  design.  Specifically,  ascertain 
that: 

1 .  The  product  and  El  of  the  technical  effort: 
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a.  Will  satisfy  the  intended  system’s  performance  requirements  and  mission  objectives 

b.  Is  in  compliance  with  security,  environmental  safety  and  occupational  health  (ES&OH) 
objectives 

c.  Is  producible,  testable,  deployable,  operable,  supportable,  maintainable,  and  reliable 

2.  The  design  and  development  process  of  the  contractor(s)  provide  for  incremental,  realistic, 
measurable  and  achievable  milestones  for  contractual  deliverables 

3.  Methodologies  are  in  place  to  identify  potential  problems  or  failures,  associated  risks  to 
program  resources  (i.e.,  cost,  schedule,  and  technical),  and  prevention  or  their  mitigation 
thereof 

This  standard  provides  the  contractor(s)  with  content  to  establish  and  maintain  a  technical  review  process 
that  meets  the  contracting  agent’s  expectations  and  intent. 

1.3  Classification 

The  following  technical  reviews  and  audits  shall  be  selected  by  the  program  manager  at  the  appropriate 
phase  of  program  development.  Each  review  and/or  audit  is  defined  in  Appendixes  A  through  J: 


SRR 

Appendix  A 

System  Requirements  Review 

SFR 

Appendix  B 

System  Functional  Review 

SAR 

Appendix  C 

Software  Requirements  and  Architectures  Review 

PDR 

Appendix  D 

Preliminary  Design  Review 

CDR 

Appendix  E 

Critical  Design  Review 

TRR 

Appendix  F 

Test  Readiness  Review 

FCA 

Appendix  G 

Functional  Configuration  Audit 

PCA 

Appendix  H 

Physical  Configuration  Audit 

SVR 

Appendix  I 

System  Verification  Review 

M/PRR  - 

Appendix  J 

Manufacturing  and  Production  Readiness  Review 

In  order  to  achieve  the  generic  technical  review  objectives  outlined  in  Section  4.0,  procedures,  planning, 
and  all  other  documentation  and  data  that  make  up  the  reviews  process  shall  be  defined  and  made 
available  to  the  contracting  agent  for  review,  assessment,  and  concurrence  at  a  mutually  agreed-upon 
time,  prior  to  the  conduct  of  each  El  review.  Roles  and  responsibilities  for  the  preparation,  conduct,  and 
acceptance  of  each  review  are  described  in  more  detail  in  Section  5. 

1.4  Application 

The  technical  reviews  and  audits  defined  herein  and  designated  by  the  contracting  agency  as  called  out  in 
the  contractual  agreements  shall  be  conducted  in  accordance  with  this  standard  to  the  extent  specified  in 
the  contract  clauses,  Statement  of  Work  (SOW),  and  the  Contract  Data  Requirements  List  (CDRL)  that 
defines  the  associated  technical  data  package  (TDP)  type,  elements  and  data  management  products 
required  at  the  appropriate  phases  of  program  development.  Scheduling  of  an  El  technical  review  and  the 
actual  timing  of  the  review  activities  shall  be  tailored  for  each  program,  using  acquisition  strategy  and 
tailoring  guidance  provided  in  Section  4  and  Appendix  K. 
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Table  1.  Acquisition  Policy  Examples  of  Specific  Technical  Review  and  Audit  Objectives 


Technical  Review  or  Audit 

Objective 

DoDI  5000.02  Phase 

Analysis  of  Alternatives 
(AoA) 

Concept  Selection,  System  CONOPS 

Material  Solution 
Analysis 

System  Requirements  Review 
(SRR) 

Review  SE  Program  Foundation  and 
Approval  of  the  Initial  Requirements 
Baseline 

Technology 

Development 

System  Functional  Review 
(SFR) 

Review  and  Approval  of  the  System 
Architecture  and  Functional 
Requirements  Baseline 

Technology 

Development 

Software  Requirements  and 
Architecture  Review 
(SAR) 

Review  and  Approval  of  the  Software 
Architecture  and  Functional 
Requirements  Baseline 

Technology 

Development 

Preliminary  Design  Review 
(PDR) 

Approval  of  the  Allocated  Baseline 

Engineering  and 

Manufacturing 

Development 

Critical  Design  Review 
(CDR) 

Approval  of  the  Design  Baseline 

Engineering  and 

Manufacturing 

Development 

Test  Readiness  Review 
(TRR) 

Verification  of  the  Contractor’s 
Readiness  to  Begin  a  Formal 

Verification  Testing 

Engineering  and 

Manufacturing 

Development 

Functional  Configuration 
Audit 
(FCA) 

Qualification  of  the  Design 

Production  and 
Deployment 

Physical  Configuration  Audit 
(PCA) 

Approval  of  the  Product  Configuration 
Baseline 

Production  and 
Deployment 

Manufacturing  Readiness 
Review 
(MRR) 

Readiness  for  Production,  Training, 
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1.5  Scheduling  of  Technical  Reviews 

The  scheduling  of  a  specific  technical  review  is  extremely  important.  If  it  is  conducted  too  early,  the  end 
item  for  review  will  not  be  adequately  defined.  Conversely,  if  the  review  is  too  late,  the  program 
commitments  could  have  been  made  erroneously,  and  correction  will  be  both  difficult  and  costly. 
Scheduling  and  planning  are  program  management  functions  and  must  be  addressed  in  detail  in  the 
negotiated  Integrated  Management  Plan  (IMP)  and  Integrated  Master  Schedule  (IMS)  after  the 
appropriate  SE  requirements  have  been  tailored  to  the  type  of  program  contracted,  e.g.: 

a.  Technology  Development  and/or  Technology  Demonstration  (TD) 
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b.  System  Development  and  Demonstration  (SDD) 

c.  Engineering  Development  (ED) 

d.  Risk  Reduction  and  Design  Development  (RRDD),  etc. 

In  accordance  with  the  practice  of  good  systems  engineering  principles,  SRR  and  SFR  shall  be  conducted 
as  “top-down”  reviews,  where  the  requirements  are  established  and  appropriately  allocated  to  the  lower 
elements  of  the  system  under  review.  Conversely,  PDR  and  CDR  shall  be  conducted  as  “bottom-up” 
reviews  from  the  lowest  elements,  culminating  in  an  overall  system  review.  SAR  typically  follows  SFR 
on  a  build-by-build  basis  for  an  individual  Software  Configuration  Item  (SWCI)  or  a  collection  of 
SWCIs. 

A  good  method  for  scheduling  technical  reviews  is  to  relate  them  to  the  documentation  requirements,  e.g., 
schedule  a  PDR  after  the  availability  of  hardware  development  specification  or  software  architecture  and 
detailed  design  and  software  test  plans,  since  the  essence  of  the  PDR  is  to  assess  the  contractor’s  approach 
to  meeting  the  requirements  of  these  documents.  Scheduling  of  technical  reviews  is  dependent  not  only 
on  documentation  availability  but  also  on  hardware  and  software  availability  and  the  completion  of  the 
acceptance  qualification  tests. 

Although  a  time  frame  for  reviews  is  defined  in  the  DoDI  5000.02  Operation  of  the  Defense  Acquisition 
System  (08  Dec  2008)  instruction,  the  review  date  for  a  given  program  may  vary.  The  initial  timing  of 
each  El  review  shall  be  provided  to  the  contracting  agency  by  the  qualified  bidders  as  part  of  their 
proposal  or  IMP  and  IMS  or  as  part  of  their  systems  engineering  management  plan  (SEMP). 
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2.  Documents 

Documents  named  or  referenced  within  this  standard,  whether  they  are  specifications,  standards, 
handbooks,  guides,  drawings,  CDRLs  or  Data  Item  Descriptions  (DIDs),  are  NOT  to  be  considered  as 
compliance  documents  unless  called  out  specifically  by  contract  Statement  of  Work  (SOW)  or 
compliance  documents  list  for  the  preparation  and  conduct  of  the  design  reviews.  The  SOW  shall  be 
referenced  for  applicable  documents.  (Copies  of  specifications,  standards,  drawings,  and  publications 
required  by  contractors  in  connection  with  specific  procurement  functions  should  be  obtained  from  the 
contracting  agency  or  as  directed  by  the  contracting  officer). 

The  following  list  of  documents  were  used  as  source  material  in  preparation  of  this  document 

1.  AFR  66-14,  Equipment  Maintenance  Policies,  Objectives,  and  Responsibilities 

2.  AFSPC  Manual  91-710  Range  SW  Safety  STD  (1  May  2004) 

3.  ANSI/EIA  649A  National  Consensus  Standard  for  Configuration  Management  (2006) 

4.  ASC/EN  GUIDE  -  USAF  Aeronautical  Systems  Center  Technical  Reviews/Audits  for  Aeronautical 
Weapon  System  Acquisition  (19  Jul  2006) 

5.  DoD  5000.4M  Cost  Analysis  Guidance  and  Procedures  (Dec  1992) 

6.  DoD  Defense  Acquisition  Guidebook  (14  October  2004) 

7.  DoD-D-lOOOB  -  Drawings,  Engineering  And  Associated  Lists  (28  October  1977) 

8.  DoD  Directive  5000.1  “The  Defense  Acquisition  System,”  (13  May  2003) 

9.  DoDI  5000.02  “Operation  of  the  Defense  Acquisition  System”  (08  Dec  2008) 

10.  DoD  MIL-STD  498  Software  Development  &  Documentation  (Dec  1994) 

11.  DoDAF  v  1.0  DoD  Architecture  Framework  (AF)  Volumes  1  and  2  (9  Feb  2004) 

12.  DoDD  4630.5  “Interoperability  and  Supportability  of  Information  Technology  (IT)  and  National 
Security  Systems  (NSS)”  (23  Apr  2007) 

13.  DoD  MIL-STD-1833  Test  Requirements  For  Ground  Equipment  And  Associated  Computer  Software 
Supporting  Space  Vehicles  (13  Nov  1989) 

14.  DoD  Risk  Management  Guide  for  Acquisition,  Sixth  Edition,  v  1.0  (Aug  2006) 

15.  DoD  Systems  Engineering  Plan  (SEP)  Preparation  Guide  Version  1.02  (10  February  2006) 

16.  DoD  Technology  Readiness  Assessment  (TRA)  Deskbook  -  DUSD(S&T)  (May  2005) 

17.  IMP  &  IMS  Preparation  and  Use  Guide  Version  0.9  (21  October  2005) 

18.  ISO/IEC  STD  15939  Software  engineering  -  Software  measurement  process  (1 1  Jul  2002) 

19.  MIL-STD-882D  DoD  Standard  Practice  for  System  Safety  (10  Feb  2000) 

20.  MIL-STD-961E  Defense  and  Program-Unique  Specifications  Format  and  Content  (Aug  2003) 

21.  MIL-STD-963B  Data  Item  Descriptions  (DIDs)  Practice  (31  August  1997) 

22.  MIL-STD- 1472F  DoD  Design  Criteria  STD  for  Human  Engineering  (23  August  1999) 

23.  MIL-STD-1521B  (USAF)  Military  Standard,  Technical  Reviews  and  Audits  for  Systems, 

Equipments,  and  Computer  Software  (4  Jun  1985)  including  Notices  1  and  2 

24.  MIL-DTL-31000C  Detail  Specification  Technical  Data  Packages  (9  Jul  2004) 

25.  SMC  Systems  Engineering  Primer  and  Handbook,  2nd  Edition  (29  Apr  2005) 

26.  SMCS-S-002  Configuration  Management  (13  June  2008) 

27.  System  and  Software  Reviews  Since  Acquisition  Reform,  Southern  California  SPIN  (2  Apr  2004) 

28.  Systems  Engineering  Plan  (SEP)  Preparation  Guide,  v  1.02  (12  Feb  2006) 

29.  TOR-2004(3909)-3537  Rev.  B  Software  Development  Standard  for  Space  Systems 
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30.  TOR-2006(8506)-5749,  Mission  Assurance  Tasks  for  Software  (30  Apr  2007) 
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3.  Definitions  of  Technical  Reviews  and  Audits 

The  terms  defined  in  this  section  are  included  to  augment  the  understanding  of  the  dialog  and  discourse 
contained  in  this  document,  by  providing  context  elaboration  and  clarification  and  consistency. 

3.1  System  Requirements  Review  (SRR) 

The  SRR  is  a  multifunctional  technical  review  or  a  series  of  reviews  that  shall  be  conducted  to  ensure  that 
all  system  and  performance  requirements  are  derived  from  the  Initial  Capabilities  Document  (ICD)  or 
draft  Capability  Development  Document  (CDD)  and  are  defined  and  consistent  with  cost  (program 
budget),  schedule  (program  schedule),  risk,  and  other  system  constraints. 

The  SRR  assesses  the  system  requirements  captured  in  the  system  specification  and  ensures  the 
consistency  between  the  system  requirements  and  the  preferred  system  solution  and  available 
technologies. 

The  SRR  is  typically  held  well  in  advance  of  Milestone  B  to  allow  time  for  issue  resolution  and  proper 
executive-level  concurrence  on  process  and  results.  The  SRR  can  convene  prior  to  program  initiation  or 
during  technology  development  and  is  typically  convened  during  system  development  and  demonstration 
when  a  significant  portion  of  the  system  functional  requirements  has  been  established. 

3.2  System  Functional  Review  (SFR) 

The  SFR  is  a  multidisciplinary  technical  review  or  a  series  of  reviews  that  shall  be  conducted,  to  ensure 
that  the  system  under  review  can  proceed  into  preliminary  design,  and  that  all  system  requirements  and 
functional  performance  requirements  derived  from  the  Capability  Development  Document  are  defined 
and  are  consistent  with  cost  (program  budget),  schedule  (program  schedule),  risk,  and  other  system 
constraints.  Generally  this  review  assesses  the  system  functional  requirements  as  captured  in  system 
specifications  (functional  baseline),  and  ensures  that  all  required  system  performance  is  fully  decomposed 
and  defined  in  the  functional  baseline.  System  performance  may  be  decomposed  and  traced  to  lower-level 
subsystem  functionality  that  may  define  hardware  and  software  requirements.  The  SFR  determines 
whether  the  system’s  functional  definition  is  fully  decomposed  to  a  low  level,  and  whether  the  Integrated 
Product  Team  (IPT)  is  prepared  to  start  preliminary  design. 

Completion  of  the  SFR  shall  provide: 

1 .  An  established  system  functional  baseline 

2.  An  updated  risk  assessment  for  the  System  Development  and  Demonstration  phase 

3.  Current  data  to  update  the  Cost  Analysis  Requirements  Description  (CARD)  document,  based 
on  the  contractor’s  proposed  system  functional  baseline 

4.  An  updated  program  development  schedule  including  system  and  software  critical  path 
drivers 

5.  An  approved  SWCI  with  updates  applicable  to  this  phase 

The  SFR  determines  whether  the  system’s  lower-level  performance  requirements  are  fully  defined  and 
consistent  with  the  mature  system  concept,  and  whether  lower-level  systems  requirements  trace  to  top- 
level  system  performance  and  the  Capability  Development  Document.  A  successful  SFR  is  predicated 
upon  the  IPTs  determination  that  the  system  performance  requirements,  lower-level  performance 
requirements,  and  plans  for  design  and  development  form  a  satisfactory  basis  for  proceeding  into 
preliminary  design. 

The  program  manager  shall  tailor  the  review  to  the  technical  scope  and  risk  of  the  system,  and  address  the 
SFR  in  the  Systems  Engineering  Plan.  The  SFR  is  the  last  review  that  ensures  that  the  system  is  credible 
and  feasible  before  more  technical  design  work  commences. 

Typical  SFR  success  criteria  include  affirmative  answers  to  the  following  exit  questions: 

1.  Can  the  system  functional  requirements,  as  disclosed,  satisfy  the  Capability  Development 
Document? 
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2.  Are  the  system  functional  requirements  sufficiently  detailed  and  understood  to  enable  system 
design  to  proceed? 

3.  Are  adequate  processes  and  metrics  in  place  for  the  program  to  succeed? 

4.  Are  the  risks  known  and  manageable  for  development? 

5.  Is  the  program  schedule  executable  (technical  and  cost  risks)? 

6.  Is  the  program  properly  staffed? 

7.  Is  the  program  with  the  approved  functional  baseline  executable  within  the  existing  budget? 

8.  Is  the  updated  Cost  Analysis  Requirements  Description  consistent  with  the  approved 
functional  baseline? 

9.  Does  the  updated  cost  estimate  fit  within  the  existing  budget? 

10.  Has  the  system  Functional  Baseline  been  established  to  enable  preliminary  design  to  proceed 
with  proper  configuration  management? 

11.  Is  the  software  functionality  in  the  approved  functional  baseline  consistent  with  the  updated 
software  metrics  and  resource  loaded  schedule? 

3.3  Software  Requirement  and  Architecture  Review  (SAR) 

The  Software  Requirements  and  Architecture  Review  (SAR)  consists  of  a  series  of  multidisciplinary 
reviews  of  the  software  requirements,  architecture,  and  test  planning  of  technical  products,  software 
development  processes,  and  the  current  state  of  the  software  development  for  all  software  items.  SAR  is  a 
review  of  the  finalized  Software  Item  (SI)  requirements  and  operational  concept.  The  SAR  shall  be 
conducted  when  SI  requirements  have  been  sufficiently  defined  to  evaluate  the  contractor’s 
responsiveness  to  and  interpretation  of  the  system,  subsystem,  or  prime  item-level  requirements.  A 
successful  SAR  is  predicated  upon  the  contracting  agency’s  determination  that  the  Software 
Requirements  Specification(s),  Interface  Requirements  Specification(s),  and  Software  Test  Plan 
Operational  Concept  Document  form  a  satisfactory  basis  for  proceeding  to  preliminary  software  design 
cycle.  See  Appendix  C  for  additional  SAR  information.  Note:  Software  Configuration  Item  (SWCI)  and 
software  item  (SI)  have  the  same  meaning  and  are  used  interchangeably  throughout  the  technical 
community. 

3.4  Preliminary  Design  Review  (PDR) 

The  PDR  is  the  formal  culmination  of  a  series  of  multidisciplinary  PDRs  of  the  system  and  of  a  series  of 
reviews  of  individual  Configuration  Items  (CIs)  that  shall  be  conducted  to  ensure  that  the  system  under 
review  can  proceed  into  detailed  design  and  can  meet  the  stated  performance  requirements  within  cost 
(program  budget),  schedule  (program  schedule),  risk,  and  other  system  constraints. 

Typical  PDR  outcomes  include: 

1.  The  assessment  that  the  system  preliminary  design  as  captured  in  performance  specifications 
for  each  configuration  item  in  the  system,  the  allocated  baseline  (ABL),  and  ensures  that  each 
function  in  the  functional  baseline  (FBL)  has  been  allocated  to  one  or  more  of  the 
configuration  items.  Configuration  items  shall  consist  of  hardware  and  software  elements  and 
include  such  items  as  airframes,  avionics,  weapons,  crew  systems,  engines,  trainers  and 
training,  etc. 

2.  The  determination  of  the  compatibility  of  the  CIs  with  performance  and  engineering  specialty 
requirements  of  the  Hardware  C  onfiguration  Item  (HWCI)  development  specification 

3.  The  evaluation  that  the  degree  of  definition  and  assess  the  technical  risk  associated  with  the 
selected  manufacturing  methods  and  processes 

4.  The  establishment  of  the  existence  and  compatibility  of  the  physical  and  functional  interfaces 
among  the  configuration  item  and  other  items  of  equipment,  facilities,  computer  software, 
and  personnel 
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5.  The  measurement  of  the  progress,  consistency,  and  technical  adequacy  of  the  selected  top- 
level  SWCI  designs  and  test  approaches 

6.  The  assessment  of  the  compatibility  between  software  requirements  and  SWCI(s)  preliminary 
design 

7.  The  PDR’s  evaluation  of  the  preliminary  version  of  the  SWCI(s)  operation  and  support 
documents 

3.5  Critical  Design  Review  (CDR) 

The  CDR  is  a  multidisciplinary  technical  review  of  the  system  and  of  a  series  of  CDRs  of  individual  CIs 
that  shall  be  conducted  for  each  configuration  item  when  detailed  design  is  essentially  complete,  with  the 
objective  to  ensure  that  the  detailed  design  of  each  individual  configuration  item  that  is  an  integral  part  of 
the  system  under  review  can  proceed  into  fabrication,  system  integration,  demonstration,  and  test,  and  can 
meet  the  stated  performance  and  engineering  specialty  requirements  of  the  configuration  item 
development  specifications  within  cost  (program  budget),  schedule  (program  schedule),  risk,  and  other 
system  constraints. 

1.  The  CDR  establishes  the  detailed  design  compatibility  among  the  configuration  item  and 
other  items  of  equipment,  facilities,  computer  software  and  personnel 

2.  The  CDR  assesses  configuration  item  risk  areas  (on  a  technical,  cost,  and  schedule  basis) 

3.  The  CDR  assesses  the  results  of  the  producibility  analyses 

4.  The  CDR  focuses  on  the  determination  of  the  acceptability  of  the  detailed  design, 
performance,  and  test  characteristics  of  the  design  solution,  and  on  the  adequacy  of  the 
operation  and  support  documents 

5.  The  CDR  assesses  the  system  final  design  as  captured  in  product  specifications  for  each 
configuration  item  in  the  system  (product  baseline),  and  ensures  that  each  product  in  the 
product  baseline  has  been  captured  in  the  detailed  design  documentation,  e.g.,  product 
specifications  for: 

a.  Hardware,  to  enable  the  fabrication  of  configuration  items,  and  include  production 
drawings 

b.  Software,  (e.g.,  software  architecture  and  detailed  design  documents)  of  one  or  more 
Software  Configuration  Item  (SWCI),  to  the  extent  specified  in  the  Software 
Development  Plan  (SDP)  based  on  the  selected  life  cycle  model(s) 

6.  For  complex  systems,  the  contractor  may  conduct  a  CDR  for  each  subsystem  or  configuration 
item.  These  individual  reviews  would  lead  to  an  overall  system  CDR.  When  individual 
reviews  have  been  conducted,  the  emphasis  of  the  overall  system  CDR  shall  focus  on 
configuration  item  functional  and  physical  interface  design,  as  well  as  overall  system  detailed 
design  requirements 

7.  The  System  CDR  determines  whether  the  hardware,  human,  and  software  final  detailed 
designs  are  complete  to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle 
model(s),  and  whether  the  contractor  is  prepared  to  start  system  fabrication,  demonstration, 
and  test 

3.6  Test  Readiness  Review  (TRR) 

The  Test  Readiness  Review  (TRR)  is  a  multifunctional  and  multidisciplinary  review  and  process 
assessment  that  shall  be  conducted,  to  verify  the  contractor’s  readiness  to  begin  a  formal  verification 
testing  event  for  an  end  item  (El).  It  is  conducted  for  each  El  to  determine  whether  the  test  procedures  are 
complete  and  to  assure  that  the  contractor  is  prepared  for  formal  El  testing.  Test  procedures  are  evaluated 
for  compliance  with  respective  test  plans  and  descriptions,  and  for  adequacy  in  accomplishing  test 
requirements.  At  TRR,  the  contracting  agency  also  reviews  the  results  of  development  testing  and  any 
updates  to  the  operation  and  support  documents.  A  successful  TRR  is  predicated  on  the  contracting 
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agency’s  determination  that  the  test  procedures,  environment,  and  previous  test  results  form  a  satisfactory 
basis  for  proceeding  to  formal  El  testing. 

3.7  Functional  Configuration  Audit  (FCA) 

The  Functional  Configuration  Audit  (FCA)  is  a  verification  process  that  shall  be  convened  periodically  to 
ascertain  that  the  actual  performance  of  a  Cl  meets  the  requirements  stated  in  its  performance 
specification  and  to  certify  that  the  Cl  has  met  those  requirements. 

1.  The  FCA  is  convened  to  verify  that  the  actual  performance  of  the  system  meets  the 
requirements  stated  in  the  system  performance  specification.  For  very  large,  complex 
Cls/systems,  the  audits  can  be  accomplished  in  increments.  Each  increment  shall  address  a 
specific  functional  area  of  the  Cl/system  and  document  any  discrepancies  that  are  found  in 
the  performance  capabilities  of  that  increment 

2.  The  FCA  also  reviews  the  completed  operation  and  support  documents 

3.  A  formal  FCA  is  performed  to  validate  that  the  development  of  a  configuration  item  has  been 
completed  satisfactorily  and  that  the  configuration  item  has  achieved  the  performance  and 
functional  characteristics  specified  in  the  functional  or  allocated  configuration  identification 

4.  A  summary  FCA  can  be  convened  to  address  the  status  of  all  of  the  action  items  that  have 
been  identified  by  the  incremental  FCAs,  in  order  to  document  and  certify  their  completion 

3.8  Physical  Configuration  Audit  (PCA) 

The  PCA  is  a  technical  examination  of  a  designated  configuration  item  that  shall  be  performed,  to  verify 
that  the  configuration  item  “As  Built”  conforms  to  the  technical  documentation  that  defines  the  actual 
configuration  of  an  item  being  produced. 

1.  The  PCA  verifies  that  the  design  documentation  matches  the  El  as  specified  in  the  contract 

2.  The  PCA  confirms  that  the  manufacturing  processes,  quality  control  system,  measurement 
and  test  equipment,  and  training  are  adequately  planned,  tracked,  and  controlled.  The  PCA 
also  validates  many  of  the  supporting  processes  used  by  the  contractor  in  the  production  of 
the  item,  and  verifies  other  elements  of  the  item  that  may  have  been  impacted  or  redesigned 
after  completion  of  the  System  Verification  Review  (SVR) 

3.  The  PCA  is  convened  prior  to  the  full-rate  production  decision  or  when  the  Government 
plans  to  control  the  detailed  design  of  the  item  it  is  acquiring  via  the  Technical  Data  Package. 
When  the  Government  does  not  plan  to  exercise  such  control  or  purchase  the  item’s 
Technical  Data  Package  (e.g.,  performance-based  procurement),  the  contractor  shall  conduct 
an  internal  PC  A  to  define  the  starting  point  for  controlling  the  detailed  design  of  the  item  and 
establishing  a  product  baseline 

4.  The  PCA  is  considered  complete  when  the  design  and  manufacturing  documentation  matches 
the  item  as  specified  in  the  contract 
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3.9  System  Verification  Review  (SVR) 

The  SVR  is  a  test,  inspection,  or  analytical  process  by  which  a  group  of  configuration  items  that  make  up 
the  system  shall  be  verified  to  have  met  specific  contracting  agency  contractual  performance  requirements 
(specifications  or  equivalent).  This  review  does  not  apply  to  hardware  or  software  requirements  verified 
at  FCA  for  the  individual  configuration  item. 

3.10  Manufacturing  Readiness  Review  (MRR) 

MRRs  shall  be  conducted  by  the  prime  contractor,  to  ensure  readiness  to  build  a  quality  product  that 
inherently  embodies  defense-unique  and/or  defense-critical  manufacturing  capabilities  characteristic  of  a 
desired  defense  contractor,  as  appropriate  for  the  program  under  review,  before  commencing  manufacture 
of  a  unit  or  other  contractually  designated  configuration  items  by  the  prime  contractor,  subcontractor,  or 
critical  component  supplier. 

3.11  Production  Readiness  Review  (PRR) 

PRRs  shall  be  conducted  to  assist  the  Government  in  the  evaluation  of  the  production  risks  and  the 
contractor’s  methodology  to  manage  those  risks  and  the  determination  that  the  contractor,  their  sub¬ 
contractors,  and  suppliers  have  resolved  to  the  satisfaction  of  the  Government  the  PRRs’  findings  and 
specific  actions  prior  to  executing  a  production  go-ahead  decision. 

1.  PRRs  are  accomplished  in  an  incremental  fashion  during  the  Full-Scale  Development  phase 
(usually  two  initial  reviews  and  one  final  review)  to  assess  the  risk  in  exercising  the 
production  go-ahead  decision.  In  its  earlier  stages  the  PRR  concerns  itself  with  gross-level 
manufacturing  concerns  such  as  the  need  for  identifying  high-risk  and  low-yield 
manufacturing  processes  or  materials  or  the  requirement  for  manufacturing  development 
effort  to  satisfy  design  requirements.  The  reviews  become  more  refined  as  the  design 
matures,  dealing  with  such  concerns  as  adequate  production  planning,  facilities  allocation, 
incorporation  of  producibility-oriented  changes,  identification  and  fabrication  of  tools  and 
test  equipment,  long-lead  item  acquisition,  etc. 

2.  The  PRR  examines  risk;  it  determines  if  production  or  production  preparations  incur 
unacceptable  risks  that  might  breach  thresholds  of  schedule,  performance,  cost,  or  other 
established  criteria 

3.  The  PRR  evaluates  the  full,  production-configured  system  to  determine  if  it  correctly  and 
completely  implements  all  system  requirements 

4.  The  PRR  determines  whether  the  traceability  of  final  system  requirements  to  the  final 
production  system  is  maintained 

5.  Timing  of  the  incremental  PRRs  is  a  function  of  program  posture  and  is  not  specifically 
locked  in  to  other  reviews 
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4.  Technical  Review  Objectives 

Technical  reviews  are  necessary  systems  engineering  activities  designed  to  assess  progress  within  a 
project  relative  to  its  planned  technical  and/or  contractual  objectives.  These  reviews: 

1.  Serve  as  a  technical  review  guide  for  the  contracting  agency  and  Government  (also 
interchangeably  referred  to  as  “contracting  agent”)  from  program  definition  through  El 
development 

2.  Assure  that  technical  reviews  are  conducted  at  logical  transition  points  in  the  development 
effort  to  assess  program  health  and  readiness  to  proceed  to  subsequent  phases  with  acceptable 
risk 

3.  Assure  that  technical  reviews  provide  a  method  for  program  teams  to  determine  progress 
achieved  to  date,  assess  predicted  or  actual  performance  against  the  requirements,  identify 
potential  problems  and  risks,  and  determine  the  need  for  mitigation  plans  to  avert  schedule 
delays  and  cost  growth  to  the  program 

4.  Assure  that  technical  reviews  facilitate  contractor  and  contracting  agent  concurrence  on 
technical  progress,  and  help  ensure  a  successful  review  against  a  mutually  agreed-to  set  of 
criteria 

5.  Assure  that  the  technical  reviews  criteria  defined  for  each  review  are  based  on  candidate 
criteria  identified  in  Section  6 

6.  Assure  that,  during  each  review,  all  criteria  elements  will  be  reviewed  against  the  program 
technical  requirements  and  their  associated  impacts  to  program  cost  and  schedule 

7.  Assure  that  the  components  of  this  document  are  tailored  by  the  contracting  agent  in  context 
of  the  acquisition  agent’s  program  objectives  and  resources  to  execute  the  end  item 
development  effort  and  take  into  account  to  the  maximum  extent  possible  the  existing 
capabilities  and  resources  that  are  already  in  place  at  the  contractor’s  and  subcontractor’s 
facilities  to  support  these  reviews 

These  objectives  are  applicable  at  all  levels  of  the  enterprise  (e.g.,  prime  contractor,  subcontractor,  and 
vendor  levels).  Government  customer  involvement  in  reviews  and  audits  shall  vary  according  to  the  needs 
of  the  specific  program. 

4.1  Reviews  Specified 

The  following  reviews  and  audits  shall  be  conducted: 


SRR 

Appendix  A 

System  Requirements  Review 

SFR 

Appendix  B 

System  Functional  Review 

SAR 

Appendix  C 

Software  Requirements  and  Architectures  Review 

PDR 

Appendix  D 

Preliminary  Design  Review 

CDR 

Appendix  E 

Critical  Design  Review 

TRR 

Appendix  F 

Test  Readiness  Review 

FCA 

Appendix  G 

Functional  Configuration  Audit 

PCA 

Appendix  FI 

Physical  Configuration  Audit 

SVR 

Appendix  I 

System  Verification  Review 

M/PRR  - 

Appendix  J 

Manufacturing  and  Production  Readiness  Review 

Technical  Review  Criteria 

Technical  Review  Criteria  (TRC)  for  all  reviews  shall  be  composed  of  all  items  necessary  to  illustrate  that 
the  program  is  technically  progressing  according  to  plan  and  ready  to  move  into  the  next  phase  or  event. 
Figure  1  illustrates  the  generic  flow  of  the  review  process  to  be  conducted  as  a  joint  activity  between  the 
Contractor  officer’s  representative  and  the  Contractor,  Subcontractor,  and  El  developer. 


Page  15  of  168 


The  list  of  TRCs  that  the  contractor  needs  to  provide  at  technical  reviews  shall  include  but  not  be  limited 
to: 

1.  A  list  of  measurable  metrics  and  accomplishments  that  demonstrate  contractor-planned 
progress  and  developmental  maturity  for  the  review  conducted 

2.  Risk  assessments  for  all  items  or  issues  that  are  on  a  critical  path,  e.g.: 

a.  No  high-risk  items  that  would  prevent  a  “move  forward”  decision 

b.  Any  high-risk  items  to  be  presented  must  be  accompanied  by  an  appropriate  mitigation 
plan 

3.  Data  that  demonstrates  performance  that  is  accumulated  and  ready  for  review 

4.  Key  decisions  that  are  complete,  executable,  and  fully  supportable 

5.  Other  items  as  specified  by  the  contracting  agent  or  as  tailored  by  contract,  prior  to  and 
specifically  for  each  review  or  audit  event 

6.  Elements  that  correspond  with  relevant  events  or  items  as  contractually  defined,  incoiporated, 
or  expanded,  i.e.: 

a.  The  program  Integrated  Master  Plan  (IMP)  and  Integrated  Master  Schedule  (IMS) 

b.  The  Statement  of  Objectives  or  Statement  of  Work  (SOW) 

c.  The  Requirements  Allocation  Document  (RAD) 

7.  Action  items  from  previous  technical  reviews  are  completed  and  closed 

The  TRC  list  is  prepared  to  ensure  that  the  reviews  culminate  in  a  determination  of  readiness,  or  a  direct 
plan  for  achieving  readiness  to  move  forward.  As  such,  qualification  for  entrance  into  the  review  implies 
that  all  criteria  elements  have  been  performed  or  adequately  addressed  for  a  given  review  milestone  and 
are  fully  ready  for  presentation  and  assessment. 

4.3  Technical  Review  Success 

Technical  review  success  is  achieved  when  all  criteria  elements  have  been  demonstrated  by  the  contractor 
to  the  satisfaction  of  the  contracting  agency,  against  a  mutually  agreed-to  set  of  “Acceptance  Criteria”  by: 

1 .  Concluding  and  verifying  that  the  end  state  of  a  criteria  item  has  been  accomplished 

2.  Correlating  each  end  state  to  a  specific  requirement(s) 

3.  Defining  the  basis  of  measurement  or  metrics  used  to  substantiate  and  confirm  that  the  end 
state  of  the  criteria  item  has  been  adequately  met 

“Acceptance  Criteria”  are  objective  evidence  of  accomplishment,  and  when  realized,  they  provide 
sufficient  evidence  to  the  contracting  agency’s  satisfaction  and  acceptance  that  demonstrates  that  the 
design  solution  selected  and  the  completion  criteria  elements  assigned  by  the  contractor  to  each  design 
review  milestone: 

1 .  Are  traceable  to  all  technical  requirements 

2.  Correspond  to  the  system  architecture 

3.  Can  be  synthesized  into  an  El  design  that  is  realistic  and  physically  producible 

4.  Can  be  implemented  to  satisfy  the  user’s  operational  needs  for  a  reasonable  cost  and 
acceptable  risk 

5.  Have  review  criteria  for  the  assessment  of  technical  progress  that: 

a.  Are  appropriately  tailored  to  reflect  pertinent  acquisition  strategy 

b.  Are  appropriately  tailored  to  reflect  major  risk  drivers  peculiar  or  unique  for  each 
program 

c.  Allow  for  the  preparation  of  documented  evidence  of  progress 
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d.  Are  sufficient  to  allow  for  an  informed  judgment  of  readiness  and  contracting  agency’s 
approval  to  proceed  to  program  execution  for  the  next  level  of  design  maturity 

General  guidance  for  the  “Acceptance  Criteria”,  generic  to  any  developmental  DoD  program,  is  provided 
in  Appendices  A  through  E  specific  to  SRR,  SFR,  SAR,  PDR,  and  CDR. 
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TIME 


Figure  1.  Generic  Technical  Review  Process  Map. 
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5.  Roles,  Responsibilities,  and  Authority 

This  section  describes  the  typical  roles,  responsibilities,  and  decision  authority  relationships  between  the 
contractor(s)  and  the  contracting  agency. 

Technical  review  planning  and  execution  responsibility  are  shared  equally  by  the  contracting  agency  or 
the  Procuring  Contracting  Officer  (PCO),  and  the  contractor  or  the  contractor’s  program  manager. 

5.1  Contractor  Participation  and  Responsibilities 

The  contractor  shall  be  responsible  for  conducting  the  Technical  Reviews  and  Audits  in  accordance  with 
the  following  requirements  except  as  amended  by  the  contract. 

5.1.1  Subcontractors  and  Suppliers 

The  contractor  shall  be  responsible  for  ensuring  that  subcontractors,  vendors,  and  suppliers  participate  in 
formal  reviews  and/or  audits,  as  appropriate. 

5.1.2  Chairperson 

The  chairperson  for  the  technical  review  shall  be  the  contractor’s  program  manager  or  his  and  her 
designee.  The  chairperson  is  responsible  for  publishing  the  meeting  agenda,  recording  and  publishing  the 
meeting  minutes,  maintaining  a  list  of  attendees,  meeting  location  selection,  and  coordination,  and  all 
administrative  items  associated  with  each  technical  review. 

5.1.3  The  Contractor  Requirements 

The  contractor  shall  be  responsible  for  conducting  the  Technical  Reviews  and  Audits  in  accordance  with 
the  following  requirements. 

The  contractors  shall  be  responsible  for  establishing  the  time,  place,  and  agenda  for  each  Review  and 
Audit  in  consonance  with  the  master  milestone  schedule,  subject  to  coordination  with  the  contracting 
agency. 

This  shall  be  accomplished  sufficiently,  well  in  advance  of  each  review  and/or  audit  to  allow  adequate 
preparation  for  the  meeting  by  both  the  contractor  and  the  contracting  agency. 

5. 1.3.1  Review,  Audit  Schedules,  and  Available  Information 

The  contractors  shall  ensure  that  each  Review  and  Audit  schedule  is  compatible  with  the  availability  of 
the  necessary  information  and  contract  articles,  e.g.,  system  engineering  data,  trade  study  results, 
producibility  analysis  results,  risk  analysis  results,  specifications,  manuals,  drawings,  reports,  hardware, 
software,  or  mock-ups. 

5.1.3.2  Review  and  Audit  Preparation 

The  contractors  shall  prepare  for  each  Review  and  Audit  in  sufficient  detail  consistent  with  the  scope  and 
magnitude  of  the  Review  and  Audit. 

5.1.3.3  Review  and  Audit  Conduct 

The  contractors  shall  designate  a  co-chairperson  for  each  Review  and  Audit.  Participating  contractor  and 
subcontractor  personnel  or  those  chosen  to  make  presentations  shall  be  prepared  to  discuss  in  technical 
detail  any  of  the  presented  material  within  the  scope  of  the  review. 

Specifically,  the  contractor’s  co-chairperson  or  his  and  her  designee  shall  be  responsible  for  the 
following: 

1 .  The  planning,  organization,  coordination,  and  delivery  of  each  review 

2.  The  preparation  and  approval  of  the  list  of  individuals  participating,  representing,  and  acting 
on  behalf  of  the  contractor  at  the  reviews 

3.  The  preparation  of  “Acceptance  Criteria”  to  be  used  for  review  and  approval  by  the  PCO 

a.  The  contractor  program  manager  must  assure  that  the  recommended  and  mutually  agreed- 
to  “Acceptance  Criteria”  can  be  demonstrated  to  initiate  the  review 

b.  The  contractor  shall  prepare  for  each  review,  supporting  data,  in  sufficient  detail  as 
objective  evidence  of  accomplishments,  consistent  with  the  scope  and  magnitude  of  the 
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review  per  the  mutually  agreed-to  “Acceptance  Criteria”  under  the  following  five  major 
categories: 

(1)  Systems  Engineering  and  Architecture  Development 

(2)  System,  Segment,  and  Subsystem  Design 

(3)  System  Verification  and  Validation 

(4)  Engineering  Disciplines  and  Specialty  Engineering 

(5)  Integrated  Technical  Risk  and  Mitigation 

4.  The  participation  by  subcontractors,  vendors,  and  suppliers  as  appropriate,  to  ensure 
presentation  of  objective  evidence  of  accomplishments  in  all  relevant  areas  of  acceptance 

5.  Technical  reviews  of  program  progress  shall  be  event  driven  and  conducted  when  the  system 
under  development  meets  the  review  “Acceptance  Criteria”  as  documented  in  the  SEP. 
Technical  reviews  shall  include  participation  by  subject  matter  experts  who  are  independent 
of  the  program  (i.e.,  peer  review),  unless  specifically  waived  by  the  SEP  approval  authority 
as  documented  in  the  SEP  and  in  agreement  with  the  program  IMP  and  IMS,  subject  to 
coordination  with  the  contracting  agency.  This  shall  be  accomplished  sufficiently  in  advance, 
to  allow  for  development  of  “Acceptance  Criteria”  for  the  review  and  adequate  preparation 
time  for  the  meeting,  by  both  the  contractor  and  the  contracting  agency 

6.  Establishing  that  each  review  schedule  is  compatible  with  the  availability  of  the  necessary 
information  and  contract  articles 

7.  Providing  the  necessary  resources  and  materials  to  perform  the  review  effectively.  This  can 
include,  to  the  extent  appropriate  and  tailored  for  the  type  and  scope  of  review  required  by 
the  contract,  e.g. : 

a.  Meeting  agenda  and  plans 

b.  Conference  room(s) 

c.  Applicable  systems  engineering  data,  e.g.,  specifications,  drawings,  manuals,  reports, 
schedules,  design  and  test  methods  and  data,  risk  analysis,  and  specialty  study  and  trade 
study  results 

d.  Architecture  products,  system  interfaces,  list  of  applicable  standards,  producibility 
analysis  results,  hardware,  software,  and  mock-ups 

e.  System  Verification  Cross  Reference  Matrix  (VCRM) 

f.  Previous  technical  review  action  items,  etc. 

g.  A  system  to  capture,  record,  track,  and  disposition  review  action  items  (AIs 

8.  Designating  a  chairperson  for  each  review.  This  person  is  responsible  for  publishing  the 
meeting  agenda,  recording,  consolidating,  and  publishing  the  meeting  minutes,  maintaining  a 
list  of  attendees,  meeting  location  selection  and  coordination,  and  all  administrative  items 

5.1.3.4  Meeting  Minutes 

The  co-chairperson  shall  provide  an  acceptable  method  to  record  input  to  official  meeting  minutes,  e.g.: 

1.  Minutes  be  recorded  as  dictated  by  the  chairperson  and  include,  as  a  minimum,  significant 
questions  and  answers,  action  items,  deviations,  conclusions,  recommended  courses  of  action 
resulting  from  presentations  or  discussions,  etc. 

2.  Conclusions  from  discussions  conducted  during  side  meetings  summarized  in  the  main 
meeting,  and  appropriate  comments  read  into  the  official  minutes 

3.  Recommendations  not  accepted,  recorded  with  the  reason  for  non-acceptance 

4.  Action  items  of  each  daily  session  summarized  daily  by  both  the  contractor  and  contracting 
agency  personnel  at  the  conclusion  of  each  day’s  session 
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5.1.3.5  Review  of  Minutes  and  Designation  of  Action 

All  action  items  shall  be  recorded  in  the  technical  review  minutes,  identifying  whether  contracting  agency 
or  contractor  action  (or  both)  is  required  for  its  agreed  resolution  and  disposition.  Minutes  shall  include  a 
recording  of  discussions  and  rationale  for  rejected  actions,  e.g.,  beyond  scope,  addressed  elsewhere 
(citing  location),  duplicate  of  action  XYZ,  etc. 

A  sample  action  item  form  is  provided  in  Table  2. 

5.1.3.6  Minutes  Publication 

The  contractor  shall  be  responsible  for  publishing  and  distributing  official  minutes. 

5.2  Procuring  Contracting  Officer  (PCO)  Participation  and  Responsibility 

5.2.1  Procuring  Contracting  Officer  (PCO)  Role 

The  Procuring  Contracting  Officer  (PCO)  or  his  and  her  designee  is  the  co-chaiiperson  for  each  Review 
and  Audit.  The  co-chairperson  acts  as  the  single  point  of  contact  between  the  contracting  agent  and  the 
contractor  and  responsible  for  approving  the  meeting  agenda  and  meeting  minutes. 

The  co-chairperson  is  also  responsible  for  the  following: 

1.  “Review  Team”  (the  reviewing  authority)  membership  selection,  i.e.: 

a.  The  “Review  Team”  may  establish  appropriate  groups  to  accomplish  all  or  parts  of  each 
review  listed  in  this  STD 

b.  The  organization  of  these  groups  is  at  the  discretion  of  the  program  manager  and/or  the 
reviewing  authority 

c.  Membership  of  the  reviewing  authority  may  include  the  acquisition  activity,  activity 
technical  leadership,  contractor  senior  management,  the  user  community,  and  subject 
matter  experts  from  outside  agencies,  as  necessary 

2.  Approving  the  review  schedule,  location,  agenda,  outline  and  content,  attendee  invitation  list 

3.  Preparing  the  measurable  metrics  for  assessment  and  rating  of  the  contractor’s 
accomplishment  and  progress  per  mutually  (contracting  agent  and  contractor)  agreed-on 
criteria 

4.  Preparing  review  evaluation  and  rating  and  issuing  the  authorization  to  proceed  with  decision 

5.  Preparing  daily  review  and  approval  of  action  items  (AIs)  to  ensure  that: 

a.  AIs  are  documented  on  the  AI  form  (Table  2) 

b.  The  action  items  reflect  all  significant  contracting  agency  inputs 

c.  Action  items  are  properly  composed 

d.  Importance  and  time-critical  aspects  are  assessed 

e.  Top-level  plan  of  action  is  defined  and  agreed  to 

f.  Responsible  agent  is  designated 

g.  Closure  dates  are  defined  and  agreed  to 

5.2.2  Review  Panel  and  Participant  Selection 

The  co-chairperson  shall  prepare,  approve,  and  provide  a  list  of  the  names,  organizations,  and  verified 
security  clearances  of  the  review  panel  and  participating  individuals  to  the  contractor  prior  to  each 
review. 
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Table  2.  Sample  Action  Item  Form 


AI  CONTROL  NUMBER: _  PROGRAM  NAME: _ 

ISSUE  DATE: _ _ 

ACTION  DESCRIPTION  and  DEFINITION:  |  Originator’s  Name  and  Org: 


ASSIGNEE: _ 

CONTRACTOR’S  ORG,  and  IPT: _ 

CATEGORY  (check):  ISSUE  CONCERN 


CRITICALITY  CLASS  (check):  1_,  2_,  or  3_ 
SECURITY  CLASSIFICATION  (check):  U_,  C_,  S_,  TS 
CRITICALITY  DESCRIPTION: 


DEPENDENCY  (P000  or  WBS): 

A- _ 

B  - _ 

C- _ 

SUSPENSE  DATE: _ 

NEED  DATE: _ 

STATUS  (check):  CANCELLED _ ,  OPEN_,  CLOSED_,  PENDING _ ,  REASSIGNED  TO  or  COMBINED  WITH 

AI: _ 

AI  RESOLUTION  and  CLOSURE  (OBJECTIVE  EVIDENCE  -  DATA,  REPORT,  ETC,): 

RATIONALE  FOR  REJECTION: _ 

COMPLETION  DATE:  ASSIGNEE  SIGNATURE: 

TECHNICAL  APPROVAL  (CHIEF  ENG)  TECHNICAL  APPROVAL  (SPO) 

SIGNATURE  &  DATE _ _  SIGNATURE  &  DATE _ 

CONTRACT  NO.  TECHNICAL  REVIEW  (circle):  STATUS  DATE: 

SRR  /  SFR  /  SAR  /  PDR  /  CDR 
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5.3  Formal  Acknowledgment 

This  standard  prescribes  the  requirements  for  conducting  Technical  Reviews  and  Audits  on  Systems, 
Equipments,  and  Computer  Software.  Official  acknowledgment  by  the  contracting  agency  of  the 
accomplishment  of  a  Review  and  Audit  is  not  to  be  interpreted  as  approval  of  statements  made  in  the 
minutes  or  of  matters  discussed  at  the  Review  and  Audit  and  does  not  relieve  the  contractor  from 
requirements  that  are  a  part  of  the  contract. 

The  Procuring  Contracting  Officer  (PCO)  provides  formal  acknowledgment  to  the  contractor  of  the 
accomplishment  of  each  review  after  receipt  of  review  minutes  and  disposition  of  the  AIs.  The 
contracting  agency  establishes  the  adequacy  of  the  contractor’s  review  performance  by  notification  of: 

1.  Approval:  to  indicate  that  the  review  was  satisfactorily  completed 

2.  Contingent  approval:  to  indicate  that  the  review  is  not  considered  accomplished  until  the 
satisfactory  completion  of  resultant  action  items,  which  may  require  submission  of  a 
mitigation  plan(s)  by  the  contractor  for  PCO  review,  negotiation,  and  approval,  in  order  to 
proceed,  or 

3.  Disapproval:  to  indicate  that  the  review  was  seriously  inadequate 

5.4  Data  Requirements  List  and  Cross  Reference 

When  this  standard  is  used  in  an  acquisition  that  incorporates  a  DD  Form  1423,  Contract  Data 
Requirements  List  (CDRL),  the  data  requirements  identified  below  shall  be  developed  as  specified  by  an 
approved  Data  Item  Description  (DD  Form  1664)  and  delivered  in  accordance  with  the  approved  CDRL 
incorporated  into  the  contract.  When  the  provisions  of  the  DoD  FAR  clause  on  data  requirements 
(currently  DoD  FAR  Supplement  52.227-7031)  are  invoked  and  the  DD  Form  1423  is  not  used,  the  data 
specified  below  shall  be  delivered  by  the  contractor  in  accordance  with  the  contract  or  purchase  order 
requirements.  Deliverable  data  required  by  this  standard  is  cited  in  the  following  paragraphs. 


Paragraph  No. 

Data  Requirement  Title 

Applicable  DID  No. 

5.1.2 

Conference  Agenda 

DI-ADMN-81249A 

5. 1.3.4 

Conference  Minutes 

DI-ADMN-81250A 

Up-to-date  DIDs  and  MIL-STDs  can  be  located  on  the  Web  at:  http://assist.daps.dla.mil/quicksearch/ 
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6.  Relevant  Technical  Review  and  Audit  Items 

6.1  Source  Authority  and  Material 

Source  authority  or  material,  that  define  specific  reviews  and  the  resources  required  to  execute  them  for 
El  acquisitions  compliance,  are  incoiporated  directly,  or  by  reference  into  the  program  execution  structure 
by  way  of  the  contract  statement  of  work  (SOW). 

6.2  ACAT  Programs  and  Contractual  Guidance 

Instructions  for  the  Operation  of  the  Defense  Acquisition  System  for  the  Acquisition  Category  (ACAT) 
program  levels  I  through  III  and  the  corollary  Milestone  Decision  Authority  (MDA)  for  each  level  is 
defined  by  DoDI  5000.02.  This  DoDI  identifies  statutory  and  regulatory  information  requirements  for  all 
milestones  and  phases,  Earned  Value  Management  (EVM)  implementation  policy,  the  statutory  and 
regulatory  policy  for  Acquisition  Program  Baselines  (APBs),  and  program  categories  with  unique 
decision  forums  or  policies.  These  policies  include  the  specific  statutory  and  regulatory  requirements 
applicable  to  information  technology  (IT)  programs,  National  Security  Systems  (NSS);  detailed  specific 
test  and  evaluation  (T&E)  procedures;  detailed  policy  for  resource  estimation,  Human  Systems 
Integration  (HSI),  acquisition  of  services,  Defense  Business  Systems  and  Systems  Engineering,  along 
with  the  administrative  and  international  policy  applicable  to  all  acquisition  programs. 

Technical  reviews  for  each  phase  of  the  acquisition  program  are  identified  as  major  milestone  events  in 
the  Integrated  Master  Plan  (IMP),  with  clearly  defined  “Acceptance  Criteria”  for  each  event.  Specific 
schedules  are  developed  jointly  by  the  contracting  agent  and  the  contractor  for  each  review  and  the 
supporting  lower- tier  review  activities  for  incorporation  into  the  contractor’s  Integrated  Master  Schedule 
(IMS). 

Technical  reviews  can  be  tailored  appropriately  (Appendix  K  -  Application  Guide  for  Tailoring  MIL- 
STD-1521)  to  suit  individual  program  scope  and  complexity.  Tailoring  or  elimination  of  reviews  shall  be 
coordinated  with  the  engineering  and  logistics  functional  discipline  leadership,  and  documented  in  the 
program’s  SEP. 

The  core  purpose  of  integrating  technical  reviews  into  the  Systems  Engineering  Plan  (SEP)  is  to  design, 
develop,  and  field  systems  that  meet  the  contractual  specifications  and  performance  requirements  of  the 
User  Agency. 

All  ACAT  programs  shall  include  in  the  SEP  the  following  essential  technical  reviews1  or  gated  events2 
(as  applicable): 

I.  Initial  Technical  Review  (ITR)1 

II.  Alternative  Systems  Review  (ASR)1 

III.  Integrated  Baseline  Review  (IBR)2 

IV.  System  Requirements  Review  (SRR)1 

V.  System  Functional  Review  (SFR)1 

VI.  Software  Requirements  and  Architecture  Review  (SAR)1 

VII.  Technology  Readiness  Assessment  (TRA)2 

VIII.  Preliminary  Design  Review  (PDR)1 

IX.  Critical  Design  Review  (CDR)1 

X.  Test  Readiness  Review  (TRR)1 

XI.  System  Verification  Review  (SVR)1 

XII.  Production  Readiness  Review  (PRR)1 

XIII.  Operational  Test  Readiness  Review  (OTRR)2 

XIV.  Physical  Configuration  Audit  (PCA)1 

XV.  In-Service  Review  (ISR)1 

Programs  need  not  conduct  technical  reviews  that  do  not  apply,  given  the  structure  of  the  program,  and/or 
where  in  the  acquisition  cycle  the  program  will  enter.  This  tailoring  shall  be  updated  as  part  of  setting  the 
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review  agenda  and  participants,  in  conjunction  with  the  program  engineers,  logisticians,  and  their 
functional  discipline  leadership. 

6.3  “Acceptance  Criteria” 

Appendices  A,  B,  C,  D,  and  E  address  “Acceptance  Criteria”  that  shall  be  given  special  consideration  by 
both  the  developing  agency  and  the  contractor,  as  objective  evidence  of  accomplishments,  for  entry  into 
and  successful  exit  from  the  SRR,  SFR,  PDR,  and  CDR  reviews.  Entrance  into  the  review  requires  that 
the  contractor  has  appropriately  addressed  the  criteria  elements  and  can  successfully  exit  from  the  review 
with  the  concomitant  implication  that  all  criteria  elements  are  properly  decomposed  to  the  satisfaction  of 
the  contracting  agency.  The  specific  “Acceptance  Criteria”  for  SRR,  SFR,  SAR,  PDR,  and  CDR  shall  be 
organized  as  delineated  in  Appendices  A,  B,  C,  D,  and  E  as  examples  of  objective  evidence  of 
accomplishments  under  the  following  five  major  categories: 

1 .  Systems  Engineering  and  Architecture  Development 

2.  System,  Segment,  and  Subsystem  Design 

3.  System  Verification  and  Validation 

4.  Engineering  Disciplines  and  Specialty  Engineering 

5.  Integrated  Technical  Risk  and  Mitigation 

The  developing  agency  and  the  contractor  shall  jointly  develop  unique  “Acceptance  Criteria”  for  the 
technical  reviews  F  through  J,  using  these  five  major  category  criteria  examples. 

6.4  Systems  Engineering  Process  Objectives 

The  systems  engineering  process  is  an  iterative  process  starting  with  requirements  analysis,  proceeding  to 
functional  (logical)  analysis  and  requirements  allocation,  and  then  to  design  solution  (synthesis).  Iteration 
occurs  via  feedback  loops,  systems  analysis,  and  control  throughout  the  systems  engineering  process  with 
top-down  recursions  and  increasing  levels  of  detailed  and  bottom-up  recursion  during  assembly  and 
integration  in  the  normal  course  of  complex  systems  development. 

Appendices  A,  B,  C,  D,  and  E  describe  in  expanded  detail  typical  systems  engineering  processes,  tasks, 
and  products  for  any  system  that  are  performed  in  preparation  and  demonstration  with  appropriately 
identified  criteria  at  the  five  core  design  reviews.  The  tasks  identified  are  to  be  performed  throughout  the 
system  life  cycle;  however,  the  detail  and  maturity  produced  by  these  tasks  in  demonstration  of 
“Acceptance  Criteria”  identified  within  will  be  highly  dependent  on  the  state  of  the  El’s  maturity  in  its 
life  cycle. 

It  is  the  joint  responsibility  of  the  contractor/developer  and  contracting  and  customer  agencies  to  integrate 
the  systems  engineering  process  with  the  design  review  process,  with  tailoring  of  compliance 
documentation,  standards,  and  reference  guides  and  handbooks  as  mutually  agreed  and  specified  by 
contract  in  accordance  with  (IAW)  Appendix  K. 

As  an  integral  and  inherent  part  of  the  design  review  processes,  the  developer  with  the  support  and 
approval  of  the  customer  can  develop,  refine,  and  update  with  an  increased  level  of  maturity  such  user 
requirements  and  objectives  as: 

a.  The  statements  of  capability  need  thresholds  and  objectives  corresponding  to  identified 
capability  gaps 

b.  The  architectural  products,  including  the  applicable  views  (Operational,  System,  and 
Technical)  as  directed  by  the  customer  IAW  the  DoD  Architecture  Framework  (DoDAF) 

c.  The  identification  of  alternative  materiel  approaches  and,  for  the  selected  material 
approaches,  alternative  operational  and  system  concepts  that  could  fill  capability  gaps  that 
offer  potential  for  further  refinement  and  subsequent  development 

d.  The  Warfighter’s  capability  gaps  in  the  Operational  Scenarios  and  Concept  of  Operations 
(CONOPS)  as  well  as  in  the  operational  and  system  needs  under  consideration  while  assuring 
consistency  between  the  CONOPS  and  the  contractor’s  Operations  Concept  (OPSCON) 

e.  The  assessment  of  the  relationship  between  capabilities  and  evolutionary  growth  in 
capabilities,  on  the  one  hand,  versus  the  life  cycle  cost,  schedule,  and  risk  for  the  materiel 
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approaches  or  system  concepts  that  could  provide  the  capabilities,  on  the  other  hand,  to 
highlight  those  capabilities  that  drive  cost,  schedule,  or  risk 

f.  The  development  of  approaches  for  transitioning  from  a  current  system,  if  any,  that  is 
ultimately  to  be  replaced,  curtailed,  or  supplemented  by  the  new  capability 

g.  The  definition  of  technology  developments  and  other  risk  mitigation  steps  for  potential  future 
action  toward  the  development  of  promising  system  concepts 

h.  Sustainment  strategies 

i.  Definition  of  the  threat  environment  (based  on  and  referenced  to  Defense  Intelligence 
Agency  (DIA)  or  Service  Technical  Intelligence  Center-approved  documents) 

j.  Operational  test  planning 

By  doing  so,  the  developer  can  identify  any  inconsistencies  between  the  system’s  requirements,  user 
needs,  or  operational  capability  requirements  process  and  arrive  at  a  baseline  system  design  configuration 
that  is  producible  and  supportable  and  meets  desired  objectives  that  for  example: 

a)  Accurately  and  completely  reflect  the  functional  and  performance  requirements  in  the 
requirements  baseline,  including  the  minimum  or  threshold  required  operational  capabilities 
consistent  with  concepts  of  operation,  system  behavior,  and  required  functionality 

b)  Accurately  model  the  system  behavior  to  include  all  sequencing,  concurrency,  and  timing 
requirements 

c)  Sufficiently  define  the  basis  for  detailed  and  precise  functions  or  logical  elements  and  their 
allocated  or  derived  performance  and  functional  requirements  at  the  next  lower  level 

d)  Decompose  requirements  to  lower  levels  so  that  each  can  be  related  to  elements  of  the 
physical  hierarchy  to  form  the  allocated  baseline,  and  the  allocation  of  the  top-level 
performance  requirements  and  design  constraints  to  the  lower  levels  is  complete 

e)  Can  include  the  relationships  to  the  physical  solution  and  be  documented  in  the  decision 
database 

f)  Can  include  the  definition  of  both  the  internal  and  external  interfaces,  and  addresses  the 
physical  implementation,  as  well  as  the  logical  issues  such  as  data  formats,  data  semantics, 
etc. 

g)  Can  be  validated  through  customer  participation  and  concurrence,  for  example: 

(1)  Comply  with  desired  system  attributes 

(2)  Allow  for  two-way  traceability  between  each  element  of  the  requirements  baseline  and 
each  element  of  the  functional  architecture 

The  design  review  and  associated  supporting  processes  and  data  thus  provide  assessment  and  validation 
with  a  high  degree  of  confidence  against  agreed-to  criteria  that  the  system  effectiveness,  life  cycle  cost, 
schedule,  risk,  and  evolutionary  growth  potential  of  the  baseline  design  are  feasible  and  affordable. 

6.5  Technology  Readiness  Assessment  (TRA) 

TRA  is  a  regulatory  information  requirement  for  all  acquisition  programs.  The  TRA  is  a  systematic, 
metrics-based  process  that  assesses  the  maturity  of  critical  technology  elements  (CTEs),  including 
sustainment  drivers.  CTE  is  considered  that  technology  or  its  application,  either  new  or  novel,  which  a 
platform  or  system  depends  on  to  meet  system  operational  threshold  requirements  in  development, 
production,  or  operation. 
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Each  TRA  shall  score  the  current  readiness  level  of  selected  system  elements,  using  defined  Technology 
Readiness  Levels  (TRLs)  and  Manufacturing  Readiness  Levels  (MRLs).  The  TRA  shall  highlight  those 
critical  technologies  (including  critical  manufacturing-related  technologies)  and  other  potential 
technology  risk  areas  that  may  adversely  affect  milestone  decision  dates  or  relevant  decision  points. 

Each  TRA  shall  be  conducted  concurrently  with  other  Technical  Reviews,  specifically  the  SRR,  PDR, 
and  the  PRR.  The  relationships  of  TRA  to  TRL,  MRL,  individual  technical  reviews,  and  milestone 
decision  points  in  the  Systems  Acquisition  cycle  are  illustrated  in  Figure  2. 
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Appendix  A 

Appendix  A  -  System  Requirements  Review  (SRR) 


10.  System  Requirements  Review  (SRR) 

The  SRR  is  a  multifunctional  and  multidisciplinary  technical  review  and  systems  engineering  (SE) 
process  assessment  that  is  used  to  verify  that  all  requirements  (operational  and  system)  contracted  for 
development  are  derived  from  the  Capability  Development  Document  (CDD)  and  as  defined: 

a.  Are  consistent  with  program  cost,  budget,  schedule,  risk,  user,  and/or  other  constraints 

b.  Are  captured  in  the  system  specification 

c.  Are  sufficiently  mature  to  proceed  into  system  architecture  trades  and  concept  definition 

d.  Are  consistent  with  available  technologies  for  the  preferred  system  solution 

e.  Can  meet  the  program  objectives  with  manageable  risk 

An  SRR  may  be  convened  several  times,  e.g.,  prior  to  program  initiation,  during  Technology 
Development  Phase  (TDP)  and  during  system  development  and  demonstration  and  tailored  to  the 
technical  scope  and  risk  of  the  system  to  be  developed. 

10.1  General 

The  SRR  shall  be  conducted  by  the  contractor  after  the  accomplishment  of  functional  analysis  and 
preliminary  requirements  allocation  (e.g.,  operational,  maintenance,  training,  Hardware  Configuration 
Items  (HWCIs),  Software  Configuration  Items  (SWCIs),  facility  CIs,  manufacturing  considerations, 
personnel  and  human  factors)  to  determine  initial  direction  and  progress  of  the  contractor’s  Systems 
Engineering  Management  effort  and  the  convergence  upon  an  optimum  and  complete  configuration.  At 
SSR,  a  significant  portion  of  the  system  requirements  has  been  established  and  baselined,  and,  depending 
upon  the  nature  and  complexity  of  the  system,  individual  SRRs  are  scheduled  for  the  segments, 
subsystems  or  CIs.  Key  SRR  elements  include  the  assessment  that: 

a.  The  majority  of  the  key  performance  parameters  (KPPs)  are  consistent  with  System 
Requirements  as  documented  by  the  CONOPS,  analysis  of  alternatives  (AoA)  results,  Initial 
Capabilities  Document  (ICD),  and  capabilities  development  document  (CDD)  and  system 
performance  specifications 

b.  The  system  requirements,  KPPs,  and  supporting  technical  performance  measures  (TPMs)  are 
defined  and  documented  in  the  requirements  allocation  document  (RAD) 

c.  Baseline  System  Requirements  are  consistent  with: 

(1)  Cost  (program  budget  and  funding  profile) 

(2)  Interoperability 

(3)  Other  specified  system  constraints 

(4)  Risk 

(5)  Schedule  (program  schedule  and  Integrated  Master  Schedule  (IMS)) 

d.  Form,  fit,  functional,  and  performance  (FFF&P)  baselines  are  established  for  each  system 
design  concept  and  Cl,  consistent  with  mission  and  user  objectives 

10.2  Purpose 

At  SRR,  the  total  Systems  Engineering  Management  activity  and  its  output  shall  be  reviewed  for 
responsiveness  to  the  Statement  of  Work,  system,  subsystem  and  Cl  requirements.  The  SRR  is  a  key 
element  of  the  pre  and  post  Systems  Acquisition  phase  of  System  Development  and  Demonstration,  to 
verify  that  the  contractor’s  derived  design  requirements  fully  satisfy  the  customer’s  requirements,  which 
result  in  a  system  that  will  satisfy  the  stakeholder  needs  and  will  include  the  following: 
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a.  Verification  that  the  contractor’s  derived  design  requirements  and  proposed  preliminary 
system  design  support  the  acquisition  program  baseline  (APB)  and  fully  implement  the 
system  and  interface  requirements  that  are  captured  in  the: 

(1)  The  Initial  Capabilities  Document  (ICD) 

(2)  The  Capabilities  Development  Document  (CDD) 

(3)  The  Systems  Engineering  Plan  (SEP) 

(4)  The  Test  and  Evaluation  Master  Plan  (TEMP) 

(5)  The  Programmatic  ES&OH  Evaluation  (PESHE) 

(6)  The  Program  Protection  Plan  (PPP) 

(7)  The  Technology  Readiness  Assessment  (TRA) 

b.  Verification  that  the  proposed  preliminary  system  design  is  consistent  with  the  user’s: 

(1)  CONOPS 

(2)  Environmental  safety  and  occupational  health  (ES&OH)  requirements 

(3)  Integrated  Support  Plan  (ISP) 

(4)  Inter-Operability  needs 

(5)  KPPs 

(6)  Operational  Architecture  (OA) 

(7)  System  Performance  Specifications 

(8)  The  system  support  and  maintenance  objectives  and  requirements 

Of  critical  importance  to  this  review  is  the  understanding  of  the  risks  inherent  in  the  contractor’s  proposed 
system  concept  and  products  and  processes,  and,  that  the  SRR  finds  the  risks  to  be  at  an  acceptable  level 
consistent  with  the  integrated  management  plan  (IMP),  integrated  master  schedule  (IMS),  and  risk 
manageable  by  the  acquisition  agency. 

10.3  Objective 

The  SRR  is  conducted  to  assess  the  adequacy  of  the  contractor’s  systems  engineering  management  plans 
and  processes,  through  review  of  the  analytical  and  design  engineering  efforts,  tailored  to  deliver  the  APB 
objective  and  to  establish  a  preliminary  functional  baseline  (BL)  for  the  system.  These  assessments  can 
include  such  analytical  and  SE  efforts  as: 

a.  Studies 

(1)  Concepts 

(2)  Designs 

(3)  Requirements  allocation 

(4)  System  interface 

(5)  Trades  (e.g.,  functions,  mission,  support,  hardware,  firmware,  software,  etc) 

b.  Analysis  and  Synthesis 

(1)  Functional  flow 

(2)  Human  factors 

(3)  Life  cycle  cost 

(4)  Logistics  support 

(5)  Manpower  requirements  and  personnel 
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(6)  Mission  and  requirements 

(7)  Program  risk 

(8)  Requirements  development 

(9)  Specialty  Engineering  (i.e.,  Hardware  and  Software  Reliability,  Maintainability, 
Producibility,  Armament  Integration,  Electromagnetic  Compatibility,  Survivability, 
and/or  Vulnerability  (Including  Nuclear),  Inspection  Methods  and  Techniques,  Energy 
Management,  Environmental  Considerations) 

(10)  System  architecture  development 

(11)  System  cost  effectiveness 

(12)  Training  and  facility  requirements 

c.  Assessments 

(1)  Industrial  base  capability  and  maturity  for  key  technologies  and  components 

(2)  Technology  maturity 

d.  Plans  and  Planning 

(1)  Configuration  management 

(2)  Data  management 

(3)  Engineering  integration 

(4)  Initial  Programmatic  ES&OH  Evaluation  (PESHE)  compliance 

(5)  Initial  test  and  evaluation 

(6)  Integrated  test 

(7)  Milestone  schedules 

(8)  Preliminary  manufacturing 

(9)  Producibility  analysis 

(10)  Specification  generation 

(11)  System  safety 

(12)  Technical  Performance  Measurement  (TPM) 

(13)  Technology  readiness  and  management 
Key  SRR  products  shall  include,  for  example: 

a.  A  comprehensive  risk  assessment  for  system  development  and  demonstration 

b.  A  preliminary  allocation  of  system  requirements  to  hardware,  human,  and  software 
subsystems 

c.  An  affirmation  that  the  system  requirements,  preferred  system  solution,  available  technology, 
and  program  resources  (funding,  schedule,  staffing,  and  processes)  form  a  satisfactory  basis 
for  proceeding  to  the  System  Development  and  Demonstration  (SDD) 

d.  An  approved  preliminary  system  performance  specification 

e.  An  approved  product  support  plan  (PSP)  with  updates 

f.  An  approved  system  development  and  demonstration  phase  systems  engineering  plan  (SEP) 
that  addresses  cost  and  critical  path  drivers 

g.  Software  components  identification  (tactical,  support,  deliverable,  nondeliverable,  etc.) 
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The  most  important  take-away  from  the  SRR  is  a  clear  understanding  of  the  impacts  that  the  defined 
requirements  will  have  on  program  development  cost,  schedule,  and  the  delivered  system  performance 
capability  in  the  context  of  the  APB  objectives. 

10.4  SRR  “Acceptance  Criteria” 

At  SRR,  all  major  program  elements  and  risk  drivers  of  the  systems  engineering  management  activities 
shall  be  considered.  A  prime  expectation  of  the  SRR  is  that  the  review  will  result  in  an  approved  technical 
baseline  that  can  be  brought  under  change  control  and  a  determination  that  baseline  can  be  implemented 
within  constraints  of  the  cost,  schedule,  and  performance  requirements.  In  preparation  for  and  scheduling 
of  an  SRR,  the  contractor  shall  demonstrate  to  the  satisfaction  of  the  contracting  agency  that: 

a.  All  applicable  engineering  activities  properly  performed  in  support  of  each  criterion 

b.  Most  and  all  of  the  SSR  criteria  elements  agreed  to  have  been  successfully  addressed 

c.  All  criteria  elements  are  properly  decomposed 

d.  The  baseline  system  requirements  are  robust,  supportable,  and  documented  to  the  satisfaction 
of  the  contracting  agency 

The  SRR  “Acceptance  Criteria”  shall  be  organized  under  the  following  five  major  categories: 

1 .  Systems  Engineering  and  Architecture  Development  (10.4.21) 

2.  System,  Segment,  and  Subsystem  Design  (10.4.2) 

3.  System  Verification  and  Validation  (10.4.3) 

4.  Engineering  Disciplines  and  Specialty  Engineering  (10.4.4) 

5.  Integrated  Technical  Risk  and  Mitigation  (10.4.5) 

This  review  shall  serve  as  objective  evidence  of  the  contractor’s  technical  effort  that  supports  the  basic 
and  agreed-to  SRR  “Acceptance  Criteria”,  e.g.: 

a)  Can  the  system  requirements,  as  disclosed,  satisfy  the  Initial  Capabilities  Document  or  draft 
Capability  Development  Document? 

b)  Are  the  system  requirements  sufficiently  detailed  and  understood  to  enable  system  functional 
definition  and  functional  decomposition? 

c)  Is  there  an  approved  system  performance  specification? 

d)  Are  adequate  processes  and  metrics  in  place  for  the  program  to  succeed? 

e)  Have  Human  Systems  Integration  requirements  been  reviewed  and  included  in  the  overall 
system  design? 

f)  Are  the  risks  known  and  manageable  for  development? 

g)  Is  the  program  schedule  executable  (technical  and/or  cost  risks)? 

h)  Is  the  program  properly  staffed? 

i)  Is  the  program  executable  within  the  existing  budget? 

j)  Does  the  updated  cost  estimate  fit  within  the  existing  budget? 

k)  Is  the  preliminary  Cost  Analysis  Requirements  Description  consistent  with  the  approved 
system  performance  specification? 

l)  Is  the  software  functionality  in  the  system  specification  consistent  with  the  software  sizing 
estimates  and  the  resource-loaded  schedule? 

m)  Did  the  Technology  Development  phase  sufficiently  reduce  development  risks? 

n)  Have  all  IMP  and  IMS  tasks  associated  with  this  review  been  successfully  closed? 
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The  following  sections  address  the  minimum,  but  not  all-inclusive,  list  of  criteria  that  shall  be 
accomplished,  as  specifically  tailored  by  contract,  along  with  all  applicable  engineering  activities  to  be 
reviewed 

10.4.1  Systems  Engineering  and  Architecture  Development 

Evidence  of  Systems  Engineering  and  Architecture  Development  requirements  maturity  criteria  examples 
at  SRR: 

A.  Systems  Engineering  and  Architecture  Development 

1.  The  system  requirements  are  complete,  clearly  stated,  feasible,  and  verifiable.  KPP  and  system 
requirements  derived  from  the  system  CONOPS,  the  system  performance  document  (SPD),  and 
the  AoA  studies,  etc.,  e.g.,  KPP  and  system  requirements  correlated  with  the  system  performance 
specifications  and  the  Interface  Control  Documents  (ICDs) 

2.  Systems  engineering  methodology  practices  are  thorough,  practical,  and  comprehensive 

3.  System  requirements  synthesized  into  conceptual  architecture  concept(s),  e.g.: 

a.  Architectural  concepts  demonstrate  level  of  program  compliance  with  system  requirements, 
e.g.,  applied  modeling  and  synthesis  methodologies  are  based  on  proven  practices 

b.  The  architectural  view(s)  is  constructed  for  each  system  (system  of  systems,  family  of 
systems,  segments,  and  subsystems)  concept(s): 

(1)  The  system  architecture  view(s)  implements  the  system  and  internal  and  external 
interface  requirements  and  contractor  operational  concepts 

(2)  The  system  architecture  view(s)  is  feasible  and  extensible 

(3)  Candidate  architectural  views  are  developed,  derived,  and  evaluated  for  each  system 
(system  of  systems,  family  of  systems,  segments,  and  subsystems)  concept(s)  in  terms  of 
form,  fit,  and  function  (FF&F)  and  KPPs 

(4)  An  operational  view  (OV)  is  developed  that  identifies  tasks  and  activities,  performance 
requirements  by  system  components  elements  and  organizational  owners  and  operators 

(5)  A  systems  view  (SV)  is  developed  that  identifies  functional  interface  requirements  by 
system  components,  elements  and  organizational  owners,  and  operators 

(6)  Critical  Technical  Standards  View  are  developed  that  define  standards  and  conventions 
that  may  be  necessary  to  implement  the  design  concept 

4.  Conceptual  system  design  solutions  (including  alternatives)  are  developed  and  assessed  in  the 
context  of  engineering  trade  space,  technical  requirements,  system  performance,  risks  (technical, 
programmatic,  schedule,  cost),  life  cycle  cost  (LCC)  and  cost  as  an  independent  variable  (CAIV) 
studies,  etc.,  e.g.: 

a.  Key  technical  and  programmatic  details  are  developed  and  derived  for  each  candidate  system 
design  solution  and  correlated  with  the  CDD,  System  Performance  Specification  and  other 
derived  requirements,  e.g.,  the  contractor’s  operational  concepts  are  consistent  with  the 
Warfighter  needs  and  concept  of  operations  (CONOPS) 

b.  Proposed  system  design  solutions  are  assessed  with  respect  to  program  performance,  cost, 
and  schedule  risks  and  mitigation  strategies 

c.  Demilitarization  and  disposal  at  end  of  life  (EOL)  are  considered  for  system  design  solutions 
(including  alternatives) 

5.  Candidate  external  system  interfaces  are  identified,  and  initial  and  conceptual  interface 
requirements  developed,  e.g.: 

a.  Critical  external  system  interfaces  are  developed  and  derived  that  identify  key  performance 
and  interface  requirements,  consistent  with  and  referenced  by  the  draft  System  Specification 
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b.  The  contractor’s  operational  concepts  are  consistent  with  the  system  and  external  interface 
requirements 

c.  Impacts  to  internal  and  external  systems  and  system  requirements  are  identified 

d.  System  and  external  interface  requirements  meet  all  contract  provisions,  including 
contractually  imposed  specifications  and  standards 

e.  System  requirements,  as  understood  and  integrated  into  system  design  solution,  satisfy  the 
initial  capabilities  document  (ICD)  or  capability  development  document  (CDD) 

6.  Preliminary  System  Requirement  Baselines  are  established  for  each  design  concept,  e.g.: 

a.  Conceptual  system  design  concepts  and  solutions  are  documented  in  terms  of  preliminary 
mission  and  system  requirement  baselines 

b.  Conceptual  system  design  concepts  and  solutions  are  documented  in  terms  of  preliminary 
functional  baselines 

c.  All  proposed  and  recommended  necessary  system  requirements  are  documented 

7.  Preliminary  life  cycle  cost  (LCC)  and  cost  as  an  independent  variable  (CAIV)  studies  are 
developed  in  support  of  each  (design)  concept,  e.g.: 

a.  LCC  and  CAIV  modeling  and  analyses  are  applied  and  correlated  for  each  design  concept, 
e.g.,  cost  models  depicting  projected  program  development,  and  operational  and  sustainment 
costs  completed,  as  well  as  projected  cost  impacts  to  other  “external”  systems 

b.  LCC  and  CAIV  methodology  is  presented  that  demonstrates  that  valid  trade  studies  were 
conducted 

8.  System  development  cost  and  schedules  are  established,  e.g.: 

a.  Appropriate  cost  models  have  been  used  to  estimate  system  development  cost  and  schedules 

b.  Realistic  system  development  cost  drivers,  such  as  complexity  and  other  parameters,  and 
assumptions  are  documented  and  have  been  used  in  validated  system  development  cost 
models  to  develop  cost  and  schedule  estimates 

c.  The  life  cycle  cost  estimate  adequately  includes  system  support 

d.  All  of  the  developmental  and  support  tasks  are  included  in  the  life  cycle  cost  estimates 

e.  Preliminary  system  development  estimates  are  supportable  and  based  on  history,  e.g.,  the 
preliminary  system  development  cost  and  schedule  estimates  have  enough  margin  to  cover 
the  estimated  risk  appropriate  at  SRR 

9.  Traceability  of  system  architecture  and  design  concept(s)  to  KPPs  is  documented  and 
demonstrated  in  an  acquisition  agency-approved  and  designated  database  and  data  repository, 
e.g.: 

a.  System  architecture(s)  and  design  concept(s)  trade  studies  are  captured 

b.  System  architecture(s)  and  design  concept(s)  have  traceability  to  KPPs  and  are  validated  by 
system  trade  studies 

c.  Preliminary  system  internal  and  external  interface  requirements  are  consistent  with  system 
interoperability  requirements  and  the  Initial  Capabilities  Document 

d.  The  system  internal  and  external  interface  requirements  are  baselined  and  are  under 
configuration  management  for  each  system  architecture  and  design  concept 

10.  The  Preliminary  System  Performance  Specification  is  developed  and  demonstrated  to  meet 
mission  requirements  for  each  system  design  concept,  e.g.,  for  each  design  concept,  evidence  is 
provided  using  models,  simulations,  analyses,  and  test  results  from  analogous  systems  to  ensure 
that  key  mission  and  performance  requirements  (CONOPS,  ICD,  and  KPPs)  are  met. 
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B.  Interoperability  Architecture  Development 

1 .  Justify  the  selected  DoD  Information  Standards  Repository  (DISR)  standards  to  meet  system  and 
mission  interoperability  requirements  for  each  system  (System  of  Systems,  Family  of  Systems, 
Segments,  and  Subsystems)  design  concept,  i.e.,  system  designs  must  be  DISR  compatible  and 
compliant 

2.  New  or  unique  standards  outside  DISR  are  submitted  for  approval  and  DISR  consideration  (e.g., 
new  data  formats,  data  exchange  protocols  and  schemas,  Ethernet  alternatives) 

3.  A  preliminary  interoperability  system  architecture  is  defined  for  each  design  concept 
implemented,  by  the  system  and  external  interface  requirements 

4.  Preliminary  interoperability  analyses  are  completed,  ensuring  compatibility  and  defining 
interrelationships  between  users  and  operators 

5.  Interoperability  trade  studies  and  requirements  analyses  are  completed,  e.g.: 

a.  Interoperability  performance  and  design  parameters  and  drivers  are  derived  from 
requirements  analysis  and  trade  studies 

b.  Results  are  integrated  into  all  system  baselines  and  models 

c.  A  methodology  is  presented  that  demonstrates  all  critical  and  major  requirements  assessed 

6.  The  preliminary  system  architecture  supports  implementation  of  operational  concepts  and 
interoperability  objectives 

7.  System  operational  concepts  include,  e.g.: 

a.  Both  nominal  and  off-nominal  scenarios  from  a  hardware  and  software  perspective,  e.g., 
processor  failover,  redundancy  management  consistent  with  the  system  architecture 

b.  Elaborated  time  lines  for  nominal  and  off-nominal  scenarios  consistent  with  the  system 
architecture 

c.  Management  of  satellite  vehicle,  constellation,  and  mission,  as  appropriate 

d.  Identification  of  operations  and  maintenance  staffing,  e.g.,  numbers,  skills,  roles,  and 
positions,  consistent  with  the  system  architecture 

8.  The  preliminary  system  architecture  fully  implements  the  system  and  external  interface  (I/F) 
requirements 

9.  The  preliminary  system  I/Fs  to  the  Global  Information  Grid  (GIG)  are  identified 

10.  Applicable  GIG  key  interface  profiles  (KIPs)  are  identified 

10.4.2  System  (System  of  Systems,  Family  of  Systems),  Segment,  and  Subsystem  Design 
Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  maturity  criteria  examples  at  SRR: 

A.  System,  Segment,  and  Subsystems  Design 

1.  The  Preliminary  System,  Segment,  and  Subsystem  is  identified;  preliminary  design  concepts  are 
established  and  major  and  critical  performance  parameters  are  defined 

2.  System,  Segment,  and  Subsystem  conceptual  design  solutions  are  assessed,  considering 
performance  requirements,  engineering  trade  space,  technology  status  and  deficiencies,  and 
technical,  programmatic,  schedule,  and  cost  risks,  e.g.: 

a.  Engineering  analysis  adequately  demonstrates  that  the  system  architecture  is  capable  of 
meeting  the  key  performance  parameters  and  driving  requirements 

b.  The  contractor’s  proposed  set  of  technical  performance  measures  (TPMs)  include  all  key 
performance  parameters  (KPPs)  and  critical  design  parameters  necessary  for  adequately 
assessing  the  evolving  system  capability 
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c.  The  contractor’s  metrics  and  TPM  processes  reflect  sound,  state-of-the-art  practices, 
techniques,  and  tools 

d.  Engineering  analysis  demonstrates  sound  use  of  engineering  principles  and  techniques 

3.  System(s)  (System  of  Systems,  Family  of  Systems,  Segments,  and  Subsystems)  performance 
parameters,  characteristics,  design  challenges,  and  risk  assessment  are  completed  and  integrated 
into  the  system  risk  model 

4.  Critical  performance  and  functional  requirements  are  included  in  individual  System(s)  (System  of 
Systems,  Family  of  Systems,  Segments,  and  Subsystems)  design  concept  solutions 

5.  System(s)  operational  sustainment  requirements  are  defined  and  derived,  e.g.: 

a.  Critical  system  performance  requirements  are  derived  and  are  traceable  to  program 
requirements  and  CONOPS 

b.  LCC  modeling  is  developed  for  each  concept  solution  with  traceable  justification 

6.  System(s)  Command,  Control,  Communication,  Computer,  and  Intelligence  (C4I)  Requirements 
Analyses  are  assessed  and  preliminary  performance  is  allocated  across  segments  and  subsystems, 

e. g.: 

a.  Preliminary  C4I  strategy  identifies  battle  management  and  information  technology  (IT)  needs 
and  dependencies  between  system(s)  (System  of  Systems,  Family  of  Systems,  Segments,  and 
Subsystems) 

b.  Preliminary  net-centric  (i.e.,  network)  interface  trade  studies  are  completed  and  candidate 
architectures  and  information  environments  defined,  including  interoperability  requirements 

c.  Preliminary  C4I  performance  requirements  ensure  interoperability,  interconnectivity, 
supportability,  synchronization,  and  sufficiency  for  each  design  concept 

7.  Threat  scenarios  are  completed  and  threat  environments  defined,  enveloped,  and  correlated  with 
system(s)  design  concepts,  e.g.: 

a.  Threat  scenarios  and  environments  are  defined  and  evaluated;  performance  parameters  are 
defined,  and  system  design  concepts  are  established  and  traceable  to  threats 

b.  Demonstrates  that  threat  operational  criteria  are  incoiporated  into  system  design  concepts 

8.  Environments  (e.g.,  natural  thermal,  humidity,  transport)  are  defined  and  performance  parameters 
derived  and  enveloped,  e.g.: 

a.  Environmental  parameters  are  derived  from  known  source  data,  system  functional  analysis 
using  proven  methodology 

b.  Environmental  parameters  are  incorporated  into  system  design  concepts 

B.  Performance  requirements  of  major  components  are  identified  and  assessed  for  each  candidate  system 
(System  of  Systems,  Family  of  Systems,  Segments,  and  Subsystems)  solution,  e.g.: 

1.  All  major  components  are  identified  based  on  system  design  concepts,  including  use  of  heritage 
systems,  components,  and  technology,  as  well  as  new  designs 

2.  Key  parameters  and  information  are  developed  and  assessed  for  each  major  component: 

a.  Major  performance  parameters  are  identified 

b.  Critical  technologies  are  identified,  including  deficiencies 

3.  Critical  design  and  manufacturing  requirements  and  challenges  are  identified,  including  COTS 
and  diminishing  manufacturing  source  (DMS) 

4.  Preliminary  reliability,  availability,  maintainability,  and  testability  requirements  are  defined  and 
design  factors  established,  e.g.: 
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a.  Critical  performance  parameters  are  defined  and  derived  from  requirements  analysis  and 
proven  methodologies 

b.  Solution  concepts  required  to  meet  performance  parameters  are  established  and  assessed 
(including  verification  and  analysis  methodologies) 

c.  Major  component  reliability,  availability,  maintainability,  and  testability  design  factors  for 
hardware  and  software  are  incorporated  into  system  design  concepts,  e.g.,  allocation  and  fault 
detection  and  isolation  capabilities  are  defined  between  elements  of  built-in  test,  fault 
detection  and  isolation  subsystem,  separate  support  equipment,  and  manual  procedures 

d.  Preliminary  technology  system  performance  requirements  are  analyzed,  concept(s)  trade 
studies  are  accomplished.  Technology  Readiness  Level  (TRL)  assessments  are  completed, 
and  a  development  strategy  is  established,  e.g. : 

(1)  TRL  demonstrated  or  technology  development  strategies  are  defined,  including  resource 
and  schedule  requirements 

(2)  Program  (Cost,  Schedule,  and  Technical)  risks  are  identified,  characterized,  and 
prioritized,  and  mitigation  strategies  and  Bum  Down  Plans  (BDP)  are  developed  and 
integrated  into  the  IMS  and  System  Risk  Model 

5.  Preliminary  Industrial  Base  (IB)  assessment  is  completed  and  risk  areas  identified  and  prioritized 
against  assessment  results,  e.g.: 

a.  IB  assessment  data  is  delineated  and  risk  areas  identified 

b.  Mitigation  strategies  are  developed,  including  resource  and  schedule  requirements 

Note:  The  following  examples  are  intended  to  provide  clarification  of  the  types  of  data  and  level  of  detail 
expected  to  be  addressed  at  SRR.  It  is  intended  that  the  contractor  will  identify  those  subsystems  and 
components  applicable  to  the  type  of  system  being  developed  and  the  appropriate  criteria  for  each 
subsystem  and  component  necessary  to  effectively  evaluate  and  assess  the  proposed  system  concept  and 
technical,  cost,  and  schedule  parameters,  e.g.: 

=>  For  Electrical  Power: 

•  Preliminary  Electrical  Power  Distribution  System  (EPDS)  performance  requirements, 
characteristics,  and  operational  criteria  are  defined,  including  initial  power  budgets,  total  power 
demand  with  allowable  margins,  and  modes  of  operation  (frequency  and  duration) 

•  Preliminary  selection  and  evaluation  of  the  type(s)  of  power  supply  sources  being  considered, 
including  their  specific  technology  and  topology 

•  Preliminary  battery  (or  energy  storage)  power  requirements  identified  and  modes  of  operation 
defined  (frequency  and  duration) 

•  Beginning-of-life  (BOL),  and  end-of-life  (EOL)  battery  life  requirements,  and  other  unique 
requirements  that  may  impact  battery  selection  or  design  are  identified 

•  Candidate  battery  cell  technologies  are  identified  and  battery  architectures  defined 
=>  For  software: 

•  Conceptual  software  architecture  is  developed,  including  modularity  structure  to  demonstrate 
software  producibility,  adaptability,  maintainability 

•  Initial  processing  capacity  and  throughput  requirements  are  established 

•  Reprogrammability  criteria  and  capability  are  defined 

•  A  preliminary  estimate  of  equivalent  source  lines  of  code  (ESLOC)  is  made 

•  Preliminary  software  risk  management  and  mitigation  processes  are  defined 
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o  System  and  program  risks  include  preliminary  critical  software  risks  as  appropriate,  e.g., 
complexity,  size,  processing  speed,  throughput,  schedules,  COTS  availability,  legacy  reuse 
suitability,  and  software  development  processes  and  tools 

o  Software  risk  management  processes  are  part  of  the  software  development  and  integrated 
with  the  System  Risk  Management  Model 

•  The  conceptual  software  architecture  addresses  the  use  of  open  systems  standards  and  satisfies  all 
appropriate  interoperability-related  requirements 

10.4.3  System,  Segment,  and  Subsystem  Verification  and  Validation 

Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  verification  and  validation  (V&V) 
requirements  maturity  criteria  examples  at  SRR: 

A.  Preliminary  System,  Segment,  and  Subsystem  V&V  strategies,  concepts,  and  methodologies  are 
developed: 

1.  Conceptual  strategies  are  established  to  verify  and  validate  system(s)  performance  requirements 
and  parameters,  e.g.: 

a.  V&V  strategy  and  methodology  for  System,  Segment,  and  Subsystem  and  component 

b.  V&V  strategy  methodology  and  techniques,  e.g.: 

(1)  Analytical,  modeling  and  simulation  (M&S),  and  testing 

(2)  Use  of  new  technology,  qualification  practices,  system(s)-level  demonstrations  and  tests 

(3)  External  organizations  and/or  facilities  and  resource  requirements  and  support 

(4)  Use  of  proven  practices 

B.  System,  Segment,  and  Subsystem  operational  functions  environments  are  identified  and  defined,  and 
are  traceable  to  operations  and  FBL  through  analysis  and  trade  studies: 

1.  Preliminary  system/ s)  V&V  test  environments  are  defined  and  are  traceable  to  system/ s) 
functions  and  specification  requirements 

2.  Demonstrates  environmental  parameters  correlated  with  V&V  strategies  and  methodology 

C.  Overall  Development,  Test,  and  Evaluation  (DT&E)  elements  are  defined  for  each  conceptual 
solution  with  rationale  for  their  selection 

D.  Preliminary  OT&E  requirements  analyses  completed  and  test  criteria  defined  traceable  to  operational 
T&E  trade  results: 

1 .  Analysis  includes  input  and  requirements  from  all  potential  stakeholders 

2.  V&V  test  requirements  are  derived  and  integrated  into  program  planning  and  design  concept 

3.  Resource  and  programmatic  requirements  and  issues  are  identified  that  may  impact  program 
technical,  cost,  or  schedule  parameters 

E.  Preliminary  test  requirements  and  results,  traceable  to  operational  requirements  via  specifications 
defined  by  the  V&V  cross-reference  matrix  (VCRM). 

F.  Risk  areas  are  identified  and  mitigation  strategies  established: 

1.  V&V  test  deficiencies,  including  those  based  on  technology  deficiencies,  are  identified  and 
characterized 

2.  Risk  mitigation  strategies  are  developed  and  integrated  into  a  system  risk  model,  including 
resource  requirements 

G.  V&V  methods  for  each  requirement  are  specified,  e.g.,  a  V&V  compliance  matrix  is  developed. 
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10.4.4  Engineering  Disciplines  and  Specialty  Engineering 

Evidence  of  Engineering  Discipline  and  Specialty  Engineering  identification  and  assessment  maturity 
criteria  examples  (categories  listed  in  A  through  R  below)  at  SRR  in  terms  of, 

1 .  Key  performance  requirements 

2.  Key  performance  parameters 

3.  Use  of  heritage  systems,  components,  and  technology 

4.  Use  of  new  designs 

A.  Parts,  Materials,  and  Processes  (PM&P) 

1 .  The  preliminary  PM&P  functional  requirements  have  been  established 

2.  Initial  assessment  of  environments  and  environmental  parameters  impacting  parts  performance 
for  each  system  design  concept  is  completed 

3.  Parts  engineering  design  strategies  are  developed  for  each  design  solution  concept,  including  risk 
assessments,  technologies,  sources  of  supply,  and  the  common  quality  levels  (i.e.,  reliability)  of 
the  parts 

4.  Identification  of  potential  long-lead  items  and  processes,  and/or  facility  needs  and  impacts 

B.  Test  and  Evaluation  (T&E) 

1.  The  preliminary  T&E  strategy  is  illustrated  with  all  test  objectives,  test  environments,  and  test 
resources  identified  to  ensure  compliance  with  design  and  specified  requirements 

2.  The  preliminary  T&E  methodology(s)  is  defined  for  all  test  approaches  addressing  all  system 
components  critical  to  verifying  system  technical  requirements  and  tailored  to  the  characteristics, 
effectivity(s),  and  margins  of  each  particular  test  item 

3.  Test  and  verification  methodology(s)  for  data  gathering,  reduction,  and  analysis  is  defined, 
including  test  environment(s),  operations  and  procedures  to  be  performed,  data  acquisition 
requirements,  documentation,  methods  of  analysis,  and  pass-fail  (i.e.,  success)  criteria 

4.  Development,  qualification,  and  acceptance  testing  of  systems,  subsystems,  components, 
assemblies,  including  NDI  and  COTS,  are  implemented,  e.g.: 

a.  The  qualification  and  verification  approach  is  tailored  to  the  characteristics  of  the  particular 
item  to  be  tested 

b.  Development  test  results  are  available  to  support  candidate  selection 

C.  Survivability  and  Vulnerability 

1.  Survivability  and  vulnerability  threat  assessments  are  performed  for  each  design  concept 
establishing  KPPs  for  each  assessed  threat  and  defining  categories  of  expected  threats,  threat 
environments,  and  their  likelihood  of  occurrence 

2.  A  set  of  survivability  characteristics  and  objectives  that  are  critical  to  the  mission  are  defined. 
Specifically: 

a.  Criticality  is  defined  in  terms  of  a  system  that  meets  operational  and  survival  objectives  as 
defined  in  the  Government’s  Initial  Capabilities  Document  (ICD)  and  System  Level  Concept 
of  Operations  (CONOPS) 

b.  Characteristics  are  expressed  in  terms  of  measurable  quantitative  parameters 

c.  Characteristics  are  amenable  to  validation  by  test  and  analysis 

d.  Characteristics  are  reflected  in  the  system’s  configuration  baseline 

e.  Survivability  characteristics  should  have  evolved  into  discernible  system  hardness  attributes 
and  system  design  criteria 
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3.  Preliminary  system  and  threat  interaction  analysis  is  performed  for  each  design  concept  to 
establish  allowable  margins  for  each  threat 

4.  Survivability  design  criteria  derived  from  threat  analyses  support  the  candidate  design  solutions 
to  mitigate  each  assessed  threat 

D.  Environmental  Safety  and  Occupational  Health  (ES&OH) 

1.  Life  cycle  environments  for  each  design  concept  are  defined  and  assessed  against  system 
requirements 

2.  Sufficient  data  is  compiled  to  complete  Key  Decision  Point  B  (KDP-B)  Programmatic 
Environmental  Safety  and  Occupational  Health  Evaluation  (PESHE)  compliance  objectives  IAW 
appropriate  standards  as  started  at  KDP-A,  including  an  assessment  of  internal  and  external 
operational  environments  for  each  design  concept 

3.  Critical  human  safety  and  health  factors  are  identified  and  incoiporated  into  the  system  safety 
program  architecture 

4.  Hazardous  materials  management  and  pollution  prevention  tasks  are  identified  and  prioritized 

E.  Mass  Properties 

1.  An  initial  mass  properties  budget  is  established,  including  mass  properties  and  appropriate 
margins 

2.  Parameters  for  weight  growth,  center  of  gravity,  and  moments  of  inertia  predictions  are 
established 

F.  System  Security  Engineering  (SSE),  Communications  Security  (COMSEC),  Information  Assurance 

(IA),  and  Program  Protection  (PP) 

1.  SSE,  IA,  COMSEC,  and  PP  security  requirements  are  identified  for  each  preferred  design  concept 
solution  in  accordance  with  DoD  and  AF  policies,  directives,  and  system  specifications  process 
and  plan  (i.e.,  DIACAP) 

2.  Integration  of  SSE,  COMSEC,  IA,  and  PP  requirements  is  defined 

3.  Security  SSE,  COMSEC,  and  PP  approaches  anti-tamper  applications,  (including  a  preliminary 
security  concept,  threat,  vulnerability  and  risk  assessments,  protection  countermeasures,  and  test 
and  evaluation  methodology  and  requirements),  are  defined 

4.  Preliminary  IA  controls  are  identified  for  system  and  data  protection,  availability,  integrity, 
confidentiality,  and  authentication,  and  non-repudiation  for  each  design  concept  is  addressed  to 
include  DIACAP  certification  and  accreditation  requirements,  e.g.: 

5.  “Net-Centric  Operations  and  Warfare  (NCOW)  Reference  Model  (RM)”  -  “NCOW  RM”  is 
presented 

6.  KIP  compliance  demonstrated 

7.  IA  compliance  demonstrated 

8.  Estimated  costs  and  schedule  objectives  are  identified  for  inclusion  in  the  program’s  baseline  for 
SSE,  COMSEC 

9.  Program  Protection  and  Information  Assurance  countermeasures  for  the  system’s  life  cycle 
activities  are  developed 

10.  Software  requirements  for  information  assurance  are  complete  and  are  appropriate,  e.g., 
information  assurance  standards  are  included  in  the  System  software  Requirements 

1 1 .  Associated  Certification  and  Accreditation  timelines  are  established 

G.  Interoperability 
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1 .  Preliminary  DoD  Information  Standards  Repository  (DISR)  standards  are  identified  that  meet  the 
system  and  mission  interoperability  requirements  (i.e.,  must  be  DISR  compatible  and  compliant) 

2.  New  and  unique  standards  outside  DISR  are  recommended  for  the  selected  design  concept 
submitted  for  approval  and  incorporation  into  DISR  (i.e.,  new  data  formats,  data  exchange 
protocols  and  schemas,  Ethernet  alternatives) 

3.  Preliminary  interoperability  architecture  requirements  are  identified 

4.  Interoperability  analysis  approaches  are  selected 

H.  Reliability  and  Maintainability  (R&M) 

1.  Preliminary  R&M  requirements  and  characteristics  are  defined  (e.g.,  mission  duration,  Ao  and 
Do,  MTBF,  MTTR,  failure  modes,  single  point  of  failure,  redundancy,  etc.) 

2.  Preliminary  R&M  analysis  is  accomplished  and  results  are  fed  into  the  overall  system  architecture 
for  each  design  solution  concept 

3.  Methodologies  for  defining  Environmental  Effects  Stress  Screening  (EESS)  are  established 

4.  Packaging,  Handling,  Storage,  and  Transportability  (PHS&T)  environmental  requirements  are 
incorporated  into  the  R&M  program 

I.  Electromagnetic  Interference  (EMI)  and  Electromagnetic  Compatibility  (EMC) 

1.  The  following  EMI  and  EMC  considerations  are  addressed  for  each  design  concept,  e.g.: 

a.  Preliminary  electromagnetic  interference  control  approaches  are  developed 

b.  Preliminary  internal  and  external  EMI  and  EMC  requirements  are  defined 

c.  Preliminary  EMI  susceptibility  requirements  and  constraints  are  identified  (i.e.,  passive 
modulation,  transmitter  Radio  Frequency  Interference  (RFI)  with  vehicle  receivers  and 
ordnance,  radiated  effects  on  power  buses,  lightning  and  surge  protection) 

d.  Preliminary  EMI  and  EMC-critical  environmental  characteristics  and  sensitive  elements 
identified 

2.  A  summary  of  all  significant  areas  addressed  in  the  EMC  Control  Plan,  including  but  not  limited 
to  program  requirements  tailoring  and  the  use  of  heritage  equipment  and  other  NDI 

3.  A  summary  of  EMC  requirements  verification  planning  to  the  unit  level 

4.  The  EMC  staffing  plan 

5.  All  risk  areas  and  risk  mitigation  closure  plans 

J.  Human  Systems  Integration  (HSI) 

1.  User  interface  hardware  and  software  requirements  for  operators,  users,  maintainers,  and 
sustainers  are  decomposed  and  derived  for  each  design  concept 

2.  Usability,  maintainability,  or  supportability  requirements  are  decomposed  and  derived  from  the 
system  requirements  for  each  design  solution 

3.  Staffing,  workload,  and  skill- level  requirements  are  decomposed  and  derived  for  each  design 
concept,  e.g.,  all  HSI-related  requirements,  standards,  and  standard  practices  flowed  down  to 
subordinate  contracting  activities 

4.  Requirements  for  HSI  are  complete  and  consistent  with  appropriate  standards,  e.g.,  description 
and  definition  of  the  end  users,  operators,  maintainers,  and  sustainers  are  coordinated  with  the 
appropriate  contracting  agency  organizations 

5.  Software  requirements  for  Human  Systems  Integration  (HSI)  are  complete  and  reference  all 
appropriate  standards  (e.g.,  MIL-STD  1472F,  DoD  Human  Computer  Interface  (HCI)  Style 
Guide,  and  SMC/AXE  Report  No.  HMRB-2001-1) 
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K.  Manufacturing  and  Producibility 

1 .  Requirements  that  will  drive  stressing  design  attributes  and  associated  specialized  manufacturing 
requirements  (caused  by  attributes  like  extreme  complexity,  multiple,  very  tight  tolerances, 
precision  assembly,  handling  of  fragile  components,  etc.)  are  identified 

2.  Manufacturing  and  producibility  plans  for  new  technologies  are  identified 

L.  Life  Cycle  Logistics 

1.  Preliminary  logistics  management  information  (LMI)  is  complete  and  validated  for  each  design 
solution  concept,  including  initial  supportability  trade  studies  and  analysis  results 

2.  System-level  design  factors,  for  selected  design  concept  are  verified  for  the  following  logistics 
elements:  design  interface,  supply  support,  test  equipment,  manpower  and  personnel,  training  and 
training  equipment,  PHS&T,  facilities,  computer  resources,  technical  data,  and  maintenance 
planning,  e.g.,  supportability  requirements  and  design  factors  are  defined  for  each  design  concept 
and  are  traceable  to  CONOPS,  ICD,  and  CDD.  Design  factors  for  the  following  logistics  elements 
are  identified 

M.  System  Safety 

1.  System  safety  requirements  are  identified  for  each  design  concept,  including  preliminary  safety 
risk  analysis  results  and  mitigation  approaches 

2.  Preliminary  hazard  analysis  is  completed  and  an  initial  list  of  safety  hazards  is  identified  for  the 
test,  operation,  and  disposal  of  each  design  solution 

3.  Critical  human  safety  and  health  factors  are  identified  and  incorporated  into  the  system  safety 
program  architecture 

4.  An  initial  hazardous  materials  list  is  compiled  and  prioritized  based  on  National  Environmental 
Policy  Act  (NEPA)  and  Occupational  Safety  and  Health  Act/Administration  (OSHA)  criteria,  i.e., 
permissible  exposure  levels  (PELs),  toxicity,  volatility,  and  transportability 

N.  Risk  Assessment 

1.  Risks  are  identified,  assessed,  controlled,  minimized,  and  accepted,  i.e.: 

a.  Technology  maturity  level  of  selected  design  approach  and  associated  mitigation  plan 

b.  Utilization  of  sole  source  items 

c.  Number  and  levels  of  proposed  redundancies 

d.  Utilization  of  prototypes,  qualification  test  articles,  test  beds,  and  number  of  test  cycles  with 
corollary  schedule  considerations 

e.  Use  and  support  of  COTS  products 

O.  Quality  Assurance 

1.  Preliminary  quality  and  product  assurance  requirements  are  defined  for  each  design  concept 

2.  Preliminary  verification,  inspection,  and  test  approaches  are  identified 

P.  Environmental  Controls 

1 .  Preliminary  operational  environmental  studies  are  completed 

2.  Preliminary  environmental  control  test  and  evaluation  strategies  are  developed 

3.  Preliminary  environmental  control  reliability  trade  studies  and  analyses  are  completed 

Q.  Software 

1.  Software  System  Requirements,  e.g.: 

a.  The  software  requirements  analysis  has  included  complete  allocation  of  functionality 
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b.  All  appropriate  interface  standards  are  included  in  the  software  requirements  (e.g.,  Space  to 
Ground  Link  Set  (SGLS)). 

c.  Software  requirements  for  dependability,  reliability,  maintainability,  and  availability  are 
complete,  based  on  the  system  requirements  analysis  and  allocations  to  software  and 
hardware 

d.  Software  requirements  for  supportability  are  complete,  based  on  the  system  requirements 
analysis  and  allocations  to  software  and  hardware 

e.  Software  safety  requirements  are  complete,  based  on  the  system  requirements  analysis  and 
allocations  to  software  and  hardware 

f.  All  appropriate  software  safety  standards  (e.g.,  EWR-127  or  AFSPC  Manual  91-710)  are 
included  in  the  system  requirements 

g.  All  appropriate  information  assurance  standards  are  included  in  the  system  requirements 

h.  Software  requirements  for  reprogrammability  are  complete  for  all  appropriate  computer 
resources 

i.  Software  requirements  for  Human  Systems  Integration  (HSI)  are  complete  and  reference  all 
appropriate  standards  (e.g.,  M1L-STD  1472F,  DoD  HCI  Style  Guide,  and  SMC/AXE  Report 
No.  HMRB-2001-1) 

j.  Software  requirements  for  interoperability  with  external  elements  are  complete  and  reference 
all  appropriate  interoperability  and  open  system  standards 

k.  Software  requirements  for  margins  are  complete  for  all  computer  resources  (e.g.,  memory  and 
storage  capacity,  processor  throughput,  and  communications  bandwidth) 

l.  Software  requirements  are  appropriately  allocated  to  COTS  products 

2.  Software  requirements  for  states  and  modes  are  defined  as  allocated  from  the  system 

requirements 

3.  Software  requirements  for  information  assurance  are  complete  and  are  appropriate,  e.g., 

information  assurance  standards  are  included  in  the  system  requirements 

4.  Operational  Concepts,  e.g.: 

a.  System  operational  concepts  include  both  nominal  and  off-nominal  scenarios  from  a  software 
perspective,  e.g.,  processor  failover,  redundancy  management 

b.  Software  requirements  for  operations,  maintenance,  and  training  needs  are  complete 

c.  System  operational  concepts  include  identification  of  operations  and  maintenance  staffing, 
e.g.,  numbers,  skills,  roles,  and  positions  from  a  software  perspective 

d.  Software  requirements  for  supportability  are  complete  and  apply  to  both  software  and 
hardware 

5.  Software  Metrics  and  Technical  Performance  Measures  are  established,  e.g.: 

a.  Preliminary  software  metrics  planning  is  sufficient  for  meeting  the  information  needs  for 
program  and  engineering  management 

b.  The  selected  TPMs  include  estimates  of  utilization  for  all  computer  resources,  e.g., 
processors,  memory,  storage,  and  input  and  output  channels,  buses  and  networks 

c.  Database  and  tools  are  selected  for  metrics  and  TPM  tracking,  trending,  and  reporting 
R.  Data  Storage  (Security,  Access,  Distribution,  and  Delivery) 

1.  Preliminary  Storage  System  Capability,  Flexibility,  and  Scalability  requirements,  e.g.: 

a.  Analysis  identifies  needed  reliability,  maintainability,  and  availability  characteristics  of 
storage  systems  environments 
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b.  Capacity,  flexibility,  and  extensibility  parameters  identified  that  support  the  expected  life  of 
the  system 

c.  Key  system  components  and  requirements  for  redundancy  identified,  e.g.,  storage  media 
hardware  and  software  capabilities  and  types 

d.  Storage  system  management  requirements  are  identified 

e.  Storage  system  operational  environments  identification  and  hardening  requirements  are  stated 

2.  Storage  System  Architecture  (SSA),  e.g.: 

a.  A  preliminary  storage  system  architecture  identifies  communications,  and  processing  capacity 
requirements 

b.  A  preliminary  storage  system  requirements  are  identified,  e.g.,  centralized  vs.  distributed 
storage;  online,  near-line,  and  offline  needs;  archive  (including  hierarchical  storage 
management,  if  appropriate),  backup,  and  restore;  and  data  replication 

c.  a  preliminary  storage  hardware  components  are  identified,  e.g.,  RAID,  Storage  Area 
Networks  (SAN),  Network  Attached  Storage  (NAS),  and  Direct  Attached  Storage  (DAS), 
consistent  with  the  SSA 

d.  a  preliminary  data  management  software  capabilities  have  been  identified,  e.g.,  automatic  file 
migration  and  transparent  file  retrieval;  migration  between  hierarchical  levels;  and  utilities  to 
report  on  media  usage,  error  detection,  and  identification 

3.  Security,  e.g.: 

a.  A  preliminary  level  of  user  integrity  (e.g.,  access  control  lists)  is  identified  that  supports 
system  requirements 

b.  A  preliminary  level  of  encryption  needed  is  identified 

c.  A  preliminary  need  for  specialized  security  capabilities,  such  as  CDS,  MLS,  and  Security 
Enclaves,  has  been  identified  and  is  included  in  the  storage  system  so  as  to  ensure  that  the 
system  requirements  are  met. 

4.  Data  Distribution  Methods,  e.g.: 

a.  Preliminary  list  of  data  receivers  has  been  identified,  e.g.,  computer  and  human  agents. 

b.  Preliminary  method(s)  of  data  distributing  data  is  identified,  e.g..  Subscribe  and  Publish,  Push 
and  Pull,  and  global  or  restricted  Web-based  access 

c.  Preliminary  data  distribution  methods  are  compatible  with  the  storage  architecture 

5.  Functionality,  e.g.: 

a.  A  preliminary  analysis  identified  the  physical  aspects  of  the  functionality  needed  to  support 
the  mission 

b.  A  preliminary  types  of  platforms  (server  and  client)  and  operating  systems  supported  are 
identified 

c.  Preliminary  data  connection  and  transport  protocols  (e.g.,  fiber  channel,  infiniband,  SWCI) 
are  identified 

d.  Preliminary  reporting  (e.g.,  usage)  and  maintenance  metrics  (e.g.,  MTBF  and  MTTR)  are 
identified 

e.  Preliminary  mapping  between  metrics  and  system-level  requirements  has  been  completed 
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10.4.5  Integrated  Technical  Risk  Management  and  Mitigation 

Evidence  of  technical  risk  management  (RM)  process  maturity  criteria  examples,  as  a  key  component  of 
an  integrated  program  (technical,  cost,  schedule,  and  performance)  RM  and  Mitigation  (RM&M)  process 
at  SRR. 

A.  Risk  identification  and  risk-ranking  strategies  encompass  such  aspects  as: 

1.  The  interrelationship  among  system  effectiveness  analysis,  technical  performance  measurement, 
intended  manufacturing  methods,  and  costs 

2.  Hardware  and  software  elements  of  the  system,  subsystem,  and  component,  including  those 
elements  provided  by  other  external  organizations  or  affected  by  application  of  design  standards, 
etc. 

3.  Inherited  hardware  or  software  use 

4.  Dependence  on  external  events  that  must  be  realized 

5.  Industrial  base,  technology  development,  engineering  skills,  and  resources 

6.  Mitigation  processes  and  procedures 

7.  Follow-on  development  and  low-rate  production 

8.  System  requirements,  preliminary  system  functional  definition,  and  functional  decomposition 
maturity  and  confidence  levels 

9.  The  dependency  of  business  development  plans  on  the  Program  IMP,  IMS  and  WBS 

10.  Program  schedule,  technical  and  funding  risk  assessment  ranking,  monitoring  and  documentation 
adequacy 

1 1 .  Schedule  and  funding  risks 

12.  Technical  risks 

13.  Risk  management  database  and  tools  for  risk  metrics  collection,  analysis,  tracking,  and  reporting 

14.  Draft  mitigation  processes  and  procedures 

15.  Comprehensive  risk  assessment  for  the  follow-on  phases 

16.  System  requirements,  as  understood  and  integrated  into  system  design  solution 

17.  Initial  Capabilities  Document  (ICD)  or  draft  Capability  Development  Document  (CDD) 

18.  System  requirements  maturity  and  confidence 

19.  Preliminary  system  functional  definition  and  functional  decomposition 
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B.  Risk  mitigation  and  reduction  strategies,  avoidance,  and  control  encompass  such  items  as: 

1.  Bum-down  plans  that  are  linked  to  dependencies  on  the  Program  IMP,  IMS  and  WBS 

2.  Continuous  risk  monitoring  and  review,  identification,  assessment,  and  ranking 

3.  Technology  and  manufacturing  readiness  level  (TRL  and  MRL)  assessments  and  metrics  that 
include: 

a.  Preliminary  (or  top  5)  program-level  risks 

b.  Preliminary  risk  mitigation  plan  formulated  for  top  risks 

c.  Requirements  risk  monitoring 

d.  Software  risk  management  of  critical  software  issues,  e.g.,  complexity,  size,  processing 
speed,  throughput,  schedules,  COTS  availability,  legacy  reuse  suitability,  and  software 
development  processes  and  tools 
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Appendix  B  -  System  Functional  Review  (SFR) 


20.  System  Functional  Review  (SFR) 

The  SFR  is  a  multidisciplinary  technical  product  and  SE  process  assessment  that  is  used  to  determine 
whether  the  system  design  concept  under  review  sufficiently  mature  to  proceed  into  preliminary  design, 
and  that  all  system  requirements  and  functional  performance  requirements  derived  from  the  Capability 
Development  Document  (CDD)  as  defined: 

a.  Are  consistent  with  program  cost  and  budget,  schedule,  risk,  user  and/or  other  constraints 

b.  Are  captured  in  system  specifications  (functional  baseline) 

c.  Are  fully  decomposed  and  defined  in  the  functional  baseline  and  traced  to  lower-level 
subsystem  and  Cl  functionality  that  may  define  hardware  and  software  requirements 

d.  Maintain  consistency  with  available  technologies  for  the  preferred  system  solution 

e.  Address  all  primary  systems  engineering  functions,  including  development,  manufacturing, 
verification,  deployment,  operations,  support,  training,  and  disposal,  including  the  risks 
inherent  in  the  contractor’s  products  and  processes 

f.  Can  meet  the  program  objectives  with  manageable  risk 

SFR  shall  be  tailored  to  address  the  technical  scope  and  risk  of  the  system  and  El,  and  validate  the 
Systems  Engineering  Plan.  The  SFR  is  the  last  review  that  ensures  that  the  system  design  concept  is 
credible  and  feasible  before  preliminary  design  commences 

20.1  General 

The  SFR  shall  be  conducted  when  the  system  definition  effort  has  proceeded  to  the  point  where: 

a.  Functional  and  performance  characteristics  of  the  system  architecture  are  defined  and  all 
optional  design  concepts  are  identified 

b.  The  System’s  form,  fit,  function,  performance  (FFFP),  and  interface  (I/F)  requirements  have 
been  optimized  and  the  associated  technical  risks  assessed 

c.  Engineering  functions  of  all  primary  systems  engineering  functions,  including  development, 
manufacturing,  verification,  deployment,  operations,  support,  training,  and  disposal  have 
been  addressed 

d.  Systems  engineering  processes  that  produced  the  technical  and  allocated  baseline  (ABL) 
requirements  and  the  engineering  planning  for  the  next  phase  of  effort  have  been  defined 

e.  Technology  maturity  issues  from  manufacturing  considerations  are  understood,  issue 
resolution  plans  are  in  place,  and  production  engineering  requirements  in  subsequent  phases 
are  identified 

Key  elements  of  the  review  shall  include  but  not  be  limited  to  addressing: 

a.  The  adequacy  of  system  and  segment  requirements  and  their  allocation  to  hardware  and 
software  items  and  personnel 

b.  The  adequacy  of  the  system  architecture  to  meet  the  system  and  segment  requirements  and 
the  user’s  Concept  of  Operations 

c.  The  adequacy  of  system  and  segment  verification  planning 

d.  The  adequacy  of  supporting  engineering  analyses 

e.  The  readiness  of  necessary  technology 

f.  The  adequacy  of  engineering  and  management  plans  and  processes 

g.  The  acceptability  of  the  remaining  system  and  program  risks 
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The  principal  goal  of  the  SFR  is  to  determine  whether  the  level  of  residual  risk  is  acceptable  for 

proceeding  with  subsequent  development  activities. 

The  SFR  addresses  all  primary  systems  engineering  functions,  including  development,  manufacturing, 

verification,  deployment,  operations,  support,  training,  and  disposal. 

20.2  Purpose 

The  SFR  is  normally  conducted  after  SRR.  The  purpose  of  the  SFR  is  to: 

a.  Assess  the  direction  and  progress  of  the  contractor’s  systems  engineering  and  management 
processes 

b.  Assess  the  contractor’s  convergence  upon  a  complete,  consistent,  and  technically  sound  set  of 
system,  segment,  and  subsystem  requirements  and  system  architecture 

The  SFR  confirms  that: 

a.  The  technology  maturity  has  been  demonstrated  and  the  risk  reduction  efforts  planned  prior 
to  the  start  of  design  have  been  completed  and  the  results  have  been  reflected  in  the  proposed 
requirements  and  allocated  baseline 

b.  The  requirements  analysis  has  progressed  to  the  point  that  the  proposed  requirements  baseline 
is  accurate  and  comprehensive  (though  perhaps  with  a  few  To  Be  Determined(s)  (TBD)s,  To 
Be  Resolved(s)  (TBR)s,  and  To  Be  Supplied(s)  (TBS)(s)) 

c.  The  preliminary  allocated  baseline  reflects  the  proposed  requirements  baseline  and  is 
balanced  with  respect  to  performance,  cost,  schedule,  risk,  and  potential  for  evolutionary 
growth 

d.  The  decision  database  supports  two-way  traceability  from  the  source  of  the  requirements 
baseline  to  the  preliminary  allocated  baseline  and  from  any  element  to  the  rationale  for  that 
element 

e.  The  assessment  that  the  evolving  allocated  baseline  can  lead  to  a  design  that  will  satisfy  the 
requirements  baseline 

f.  The  preliminary  physical  hierarchy,  the  planned  or  approved  PWBS,  and  the  CWBS  that  are 
in  place  or  are  proposed  to  be  used  subsequent  to  the  SFR  are  all  consistent 

g.  The  life  cycle  cost  for  the  evolving  design  is  consistent  with  the  program  affordability 
constraints 

h.  The  remaining  risks  have  been  identified  and  can  be  handled  in  the  context  of  the  planned 
contract  and  program  activities 

20.3  Objective 

The  objective  of  the  SFR  is  to: 

a.  Evaluate  the  optimization,  correlation,  completeness,  and  risks  associated  with  the  allocated 
technical  requirements 

b.  Assess  the  systems  engineering  process  that  produced  the  allocated  technical  requirements 

c.  Evaluate  the  engineering  planning  for  the  next  phase  of  effort 

d.  Review  that  basic  manufacturing  considerations  are  identified 

e.  Verify  that  planning  for  production  engineering  in  subsequent  phases  is  addressed 

A  successful  SFR  shall  provide  as  an  example: 

a.  An  established  system  functional  baseline 

b.  An  updated  risk  assessment  for  the  System  Development  and  Demonstration  (SDD) 
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c.  An  updated  program  development  schedule  including  system  and  software  critical  path 
drivers 

d.  An  approved  SWCI  with  updates 

The  most  important  takeaway  from  the  SFR  is  a  clear  understanding  and  agreement  that: 

a.  The  lower-level  performance  requirements  of  the  system(s)  under  review  are  fully  defined 
and  are  consistent  with  the  mature  system  concept 

b.  The  lower-level  systems  requirements  trace  to  top-level  system  performance  and  the 
Capability  Development  Document 

c.  The  determination  that  the  system  performance  requirements,  lower-level  performance 
requirements,  and  plans  for  design  and  development  form  a  satisfactory  basis  for  proceeding 
to  preliminary  design 

20.4  SFR  “Acceptance  Criteria” 

At  SFR  all  major  systems  engineering  management  elements  and  activities  that  are  program  risk  drivers 
are  considered.  The  intent  of  the  SFR  is  to  ascertain  that: 

a.  An  accurate  and  comprehensive  requirements  baseline  (RBL)  can  be  approved 

b.  A  final  functional  baseline  (FBL)  can  be  established 

c.  Effective  and  efficient  progress  is  made  towards: 

(1)  Meeting  all  technical  performance  requirements 

(2)  Buying  down  risk  by  the  application  and  utilization  of  mature  technologies 

(3)  The  capability  to  track,  monitor,  status  and  achieve  TPMs 

(4)  Associated  cost  and  schedule  objectives 

Each  criterion  shall  be  deemed  successfully  accomplished  upon  demonstrating  to  the  satisfaction  of  the 
contracting  agency  that  all  applicable  engineering  activities  have  been  properly  conducted  in  support  of 
the  criterion.  Entrance  into  the  review  requires  that  the  contractor  appropriately  address  the  requirements 
criteria  elements,  and  demonstrate  a  viable  technical  and  program  risk  management  strategy.  Successful 
exit  from  the  review  implies  that  all  criteria  elements  have  been  demonstrated  to  the  satisfaction  of  the 
contracting  agency. 

The  SFR  “Acceptance  Criteria”  shall  be  organized  under  the  following  five  major  categories: 

1.  Systems  Engineering  and  Architecture  Development  (20.4. 1) 

2.  System,  Segment,  and  Subsystem  Design  (20.4.2) 

3.  System  Verification  and  Validation  (20.4.3) 

4.  Engineering  Disciplines  and  Specialty  Engineering  (20.4.4) 

5.  Integrated  Technical  Risk  and  Mitigation  (20.4.5) 

This  review  shall  serve  as  objective  evidence  of  the  contractor’s  technical  effort  that  supports  the  basic 
and  agreed-to  SFR  “Acceptance  Criteria,”  e.g.: 

a.  The  system  functional  requirements,  as  disclosed,  satisfy  the  Capability  Development 
Document. 

b.  The  system  functional  requirements  are  sufficiently  detailed  and  understood  to  enable  system 
design  to  proceed 

c.  The  processes  and  metrics  in  place  for  the  program  to  succeed  are  adequate 

d.  The  risks  are  known  and  manageable  for  development 

e.  The  program  schedule  is  executable  (technical,  cost,  and  schedule  risks) 
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f.  The  program  is  properly  staffed 

g.  The  program  with  the  approved  functional  baseline  is  executable  within  the  existing  budget 

h.  The  contractor’s  proposed  system  functional  baseline  is  consistent  with  the  current  update  of 
the  Cost  Analysis  Requirements  Description  (CARD) 

i.  The  updated  cost  estimate  fits  within  the  existing  budget 

j.  The  system  Functional  Baseline  is  established  to  enable  preliminary  design  to  proceed  with 
proper  configuration  management 

k.  The  software  functionality  in  the  approved  functional  baseline  is  consistent  with  the  updated 
software  metrics  and  resource  loaded  schedule 

l.  All  IMP  and  IMS  tasks  associated  with  this  review  have  been  successfully  closed 

The  following  sections  address  the  minimum,  but  not  all-inclusive,  list  of  criteria  that  shall  be 
accomplished  in  support  of  the  SFR,  as  specifically  tailored  by  contract,  along  with  all  applicable 
engineering  activities  to  be  reviewed. 

20.4.1  Systems  Engineering  and  Architecture  Development 

Evidence  of  Systems  Engineering,  Architecture  Development,  and  Design  Development  maturity  criteria 
examples  at  SFR: 

A.  The  system,  segment,  and  subsystem  functional  requirements  are  complete,  feasible,  verifiable,  and 
clearly  stated 

B.  KPPs,  system  requirements,  CONOPS,  and  the  SPD  are  compared  and  correlated  with  the  selected 
design  concept  and  captured  in  the  Requirements  Allocation  Document  (RAD) 

C.  Baseline  system  functional  requirements  are  derived  from  the  architecture  for  the  selected  design 
concept,  e.g.: 

1.  The  architecture  demonstrates  the  expected  level  of  program  compliance,  e.g.: 

a.  Baseline  system  modeling  and  selected  synthesis  methodology  are  based  on  proven  practices 

b.  The  system,  segment,  and  subsystem  functional  requirements  (i.e.,  internal  and  external)  are 
under  configuration  management  and  are  sufficiently  mature  to  allow  to  proceed  with 
preliminary  design 

2.  Architectural  views  for  the  selected  System,  System  of  Systems,  and  Family  of  Systems  design 

concept  are  clearly  traceable  to  derived  KPPs,  e.g.: 

a.  The  system  architecture  fully  implements  the  selected  design  concept’s  system,  segment, 
subsystem,  and  interface  requirements  and  the  contractor’s  operational  concepts 

b.  The  architecture  for  the  selected  design  concept  is  feasible  and  extensible 

3.  An  architectural  view(s)  for  the  selected  design  concept  is  established,  e.g.: 

a.  The  necessary  system  view(s)  is  determined  and  established,  correlating  systems  and 
characteristics  to  operational  needs 

b.  The  necessary  operational  view(s)  is  determined  and  established,  which  identifies  baseline 
functional  performance  requirements  by  system  components  and  elements  and  by 
organizational  owners  and  operators 

c.  The  necessary  technical  standards  view(s)  is  determined  and  completed,  establishing  the 
standards  definitions  and  conventions  necessary  to  implement  the  selected  design  solution 
concept 
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D.  The  system  design  concept  was  selected  in  the  context  of  engineering  trade  space,  technical 
requirements,  system  performance,  risks  (technical,  programmatic,  schedule,  cost),  LCC  and  CAIV 
trade  analysis,  etc.,  e.g.: 

1 .  The  key  technical  and  programmatic  details  developed  and  derived  for  the  selected  system  design 
concept,  e.g.,  the  contractor’s  operational  concepts  fully  implement  and  are  consistent  with  the 
user’s  updated  Concept  of  Operations  (CONOPS)  for  the  full  program  life  cycle 

2.  The  contractor’s  updated  operational  concepts  are  consistent  with  the  system  and  segment,  and 
interface  requirements  are  baselined  and  under  configuration  management 

3.  Interoperability  functional  requirements  are  defined  for  the  proposed  system  design  concept,  and 
performance  allocations  are  established,  e.g.: 

a.  Final  interoperability  performance  and  design  parameters  and  drivers  for  the  selected  design 
concept  are  shown  to  be  derived  from  the  requirements  analysis 

b.  Results  of  trade  studies  are  shown  to  be  integrated  into  the  selected  system  design  baseline 
and  model 

c.  Demonstrated  interoperability  requirements  and  functional  performance  criteria  are 
incorporated  into  the  selected  design  concept 

d.  Requirements  for  demilitarization  and  disposal  at  EOL  are  integrated  into  the  system  design 
baseline 

E.  System  external  interfaces  are  identified  and  functional  performance  interface  (both  internal  and 
external)  requirements  are  developed  for  the  selected  design  concept,  e.g.: 

1.  System-to-system,  segment-to-segment,  subsystem-to-subsystem  and  component-to-component 
functional  interface  analyses  are  completed 

2.  Intersegment  and  intersubsystem  interface  requirements  are  consistent  with  and  referenced  by  the 
System  Performance  and  Segment  and  Subsystem  Specifications 

3.  Impacts  to  internal  and  external  systems  and  system  requirements  are  identified  for  the  selected 
design  concept 

4.  The  system  and  external  interface  functional  requirements  meet  all  contract  provisions,  including 
compliant  specifications  and  standards  for  the  selected  design  concept,  e.g.: 

a.  The  segment,  intersegment,  subsystem,  and  intersubsystem  requirements  meet  all  compliant 
specifications  and  standards 

b.  Preliminary  Functional  Flow  Block  Diagrams  (FFBDs)  developed  demonstrate  the  flowdown 
and  traceability  between  higher  and  lower-level  requirements 

F.  Mission  and  System  Functional  Requirements  Baselines  are  established,  based  on  the  selected  design 
concept,  e.g.: 

1.  Sufficient  verified  technical  information  exists  that  supports  the  establishment  of  mission  and 
system  requirements  baselines 

2.  FBL,  based  on  the  design  concept,  is  selected;  details  adequate  to  address  all  KPP  and  system 
performance  and  specification  requirements,  e.g.,  requirements  flowdown  trade  studies  and 
analyses  from  system  to  segment  to  subsystem  to  components  are  complete  and  traceable  to 
accepted  trade  results 

G.  The  life  cycle  cost  (LCC)  and  cost  as  an  independent  variable  (CAIV)  assessment  supports  the 
selected  design  concept  and  the  functional  baseline,  e.g.: 

1.  LCC  and  CAIV  modeling  and  analyses  applied  and  correlated  with  the  selected  design  concept 
and  the  functional  baseline,  e.g.,  cost  models  representing  projected  program  development, 
operational  and  sustainment  costs  are  completed,  including  projected  cost  impacts  to  other 
“external”  systems 
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H.  Traceability  of  the  selected  design  concept  to  system  KPPs  and  system  trade  studies  has  been  clearly 
shown,  e.g.: 

1.  Traceability  of  and  correlation  to  the  selected  design  concept  to  KPPs  and  trade  study  analysis 
results  are  demonstrated 

2.  The  system  requirements  and  external  interface  functional  requirements  are  traceable  to  and  fully 
implement  the  contract  System  Performance  Specification,  segment,  and  subsystem  specifications 
and  the  user’s  Capability  Development  Document  (CDD),  e.g.,  the  segment,  intersegment, 
subsystem,  and  intersubsystem  interface  requirements  are  traceable  to  and  implement  the  system 
and  external  interface  functional  requirements 

3.  The  system  and  external  interface  functional  requirements  for  the  selected  design  concept  is 
baselined  and  under  configuration  management 

I.  The  System  Performance  Specification  developed  for  the  selected  design  concept  is  traceable  to  both 
the  requirements  and  functional  baselines,  e.g.: 

1.  Segment  and  subsystem  specifications  preliminarily  are  defined  and  traceable  to  the  System 
Performance  Specification 

2.  The  modeling  and  simulation  capability  planning  and  scheduling  is  synchronized  with  system, 
segment,  subsystem,  and  interface  design  development  plans  and  schedules 

3.  System- level  verification  cross-reference  preliminarily  is  defined  and  traceable  to  the  verification 
methodology  of  the  selected  design  concept 

J.  System  integration  and  verification  requirements  analyses  are  completed  for  the  selected  design 
concept,  e.g.: 

1.  System-level  verification  planning  completed  with  rationale  for  verification  objectives,  types, 
levels,  and  sequence  of  verification  and  verification  data  to  be  collected 

K.  A  preliminary  allocation  of  the  technical  and  functional  baselines  are  defined  for  the  selected  design 
concept,  e.g.: 

1.  Preliminary  allocation  baseline  for  the  selected  design  concept  system,  its  segments,  and 
subsystems  are  correlated  with  engineering  trade  assessments  and  risk  study  results 

2.  All  preliminary  Hardware  Configuration  Item  (HWCI)  and  Software  Configuration  Item  (SWCI) 
descriptions  are  developed 

3.  All  preliminary  HWCIs  and  SWCIs  specification  requirements  are  defined  and  are  traceable  to 
the  system  performance  specification 

4.  All  software  components  (tactical,  support,  deliverable,  nondeliverable,  etc.)  are  preliminarily 
defined  and  traceable  to  the  system  performance  specification 

20.4,2  System,  Segment,  and  Subsystem  Design 

Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  maturity  criteria  examples  at  SFR: 

A.  System,  segments,  and  subsystems  are  established  for  the  selected  design  concept  and  major  and 
critical  performance  parameters  are  baselined 

1.  The  selected  design  concept  demonstrates  traceability  among  all  considerations,  e.g.: 

a.  Performance  requirements 

b.  Engineering  trade  space,  technology  status  and  deficiencies,  and  technical,  programmatic, 
schedule  and  cost  risks 

c.  The  adequacy  of  the  selected  design  concept  has  been  demonstrated  using  engineering 
analysis,  including  all  relevant  specialty  engineering  disciplines  and  is  consistent  with  the 
TPMs  and  KPPs,  e.g.: 
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(1)  Engineering  analysis  adequately  demonstrated  that  the  allocation  of  functional 
requirements  to  hardware  and  software  items  and  personnel  will  meet  system 
requirements 

(2)  Engineering  analysis  adequately  demonstrated  the  readiness  of  the  design  to  proceed 

2.  System  performance  parameters,  characteristics,  design  challenges,  and  risk  assessments  that 
form  the  system  risk  model  for  the  selected  design  concept  are  under  configuration  management 

3.  Demonstrate  that  critical  performance  and  functional  requirements  are  incorporated  in  the 
selected  design  concept 

B.  C4I  requirements  analysis  results  and  allocation  across  subsystem  and  components  are  completed  and 
traceable  to  the  selected  system  design  concept,  e.g.: 

1.  C4I  strategy  for  the  selected  design  concept  solution  for  battle  management  and  information 
technology  (IT)  needs  and  dependencies  between  system  subsystems  and  the  system,  system  of 
systems,  and  family  of  systems 

2.  Net-centric  (i.e.,  network)  trade  study  results  demonstrated  the  architecture  and  information 
environments  for  the  selected  design  concept 

3.  Demonstrate  that  C4I  requirements  ensure  interoperability,  interconnectivity,  supportability, 
synchronization,  and  sufficiency,  e.g.,  performance  criteria  are  incoiporated  in  the  selected 
system  design  concept  and  allocated  across  segments,  subsystems  and  components 

C.  Threat  scenarios  and  threat  environments  initially  defined  or  enveloped  and  correlated  with  the 
selected  system  design  concept,  e.g.: 

1.  Threat  scenarios  and  environments  are  defined  and  validated 

2.  Performance  parameters  are  defined  and  are  traceable  to  the  selected  system  design  concept  and 
to  known  and  identified  threats 

3.  Demonstrate  that  threat  scenario  operational  and  environmental  criteria  are  incorporated  into  the 
selected  system  design  concept  and  allocated  to  segments,  subsystems  and  components 

D.  Environments  (e.g.,  natural  and  debris,  shock,  vibration,  thermal,  humidity,  vacuum)  are  defined  and 
parameters  correlated  to  the  selected  system  design  concept,  e.g.: 

1.  Environmental  parameters  derived  from  known  source  (i.e.,  similar  systems)  data  and  system 
functional  analyses  using  proven  methodology,  e.g.,  environmental  models  and  simulations 
validated 

2.  Demonstrate  that  environmental  parameters  are  incoiporated  into  the  selected  system  design 
concept  and  allocated  to  segments,  subsystems  and  components 

E.  Reliability,  availability,  maintainability,  and  testability  (RAM&T)  requirements  and  design  factors 
correlated  with  the  selected  design  concept,  e.g.,  demonstrate  that  the  RAM&T  design  criteria  are 
incorporated  into  the  selected  design  concept  and  allocated  to  segments,  subsystems  and  components 

F.  The  system  operational  sustainment  strategy  is  defined,  including  key  performance  parameters; 
design  drivers  for  sustainment  are  integrated  into  the  selected  design  concept,  including  all  major 
system  and  program  requirements,  e.g.: 

1.  Sustainment  trade  study  results  used  to  baseline  crucial  system  performance  requirements  for  the 
selected  system  design  concept  are  traceable  to  program  requirements  and  CONOPS 

2.  LCC  sustainment  model  correlates  to  the  selected  design  concept 

G.  The  development  strategy  to  achieve  the  selected  design  concept  Technology  Readiness  Levels 
(TRLs)  has  been  implemented,  e.g.,  risk  mitigation  strategies  have  been  developed  and  integrated  into 
the  system  risk  model,  including  resource  requirements  for  the  selected  design  concept 
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H.  Industrial  Base  (IB)  assessment  results  are  correlated  with  the  selected  system  design  concept,  risk 
areas  are  identified  and  prioritized  and  mitigation  strategies  defined,  e.g.: 

1.  IB  assessment  data  correlated  with  identified  and  implicit  risk  areas 

2.  Mitigation  strategies  planned  and  implemented,  including  resources  and  schedule  requirements 

I.  Performance  requirements  for  the  selected  system  design  concept’s  major  subsystems  and 
components  are  baselined: 

1.  All  major  subsystems  and  components  traceable  to  the  selected  system  design  concept  and 
identifies  use  of  heritage  systems,  components,  and  technology  as  well  as  use  of  other  new 
designs 

2.  Key  parameters  and  information  are  developed  and  assessed  for  each  major  subsystem  and 
component  of  the  selected  system  design  concept,  e.g.: 

a.  Major  performance  parameters  are  identified 

b.  Critical  technologies  are  identified,  including  deficiencies 

c.  Critical  design  and  manufacturing  requirements  and  challenges  are  identified 

Note:  The  following  examples  are  intended  to  provide  clarification  of  the  types  of  data  and  level  of  detail 
expected  to  be  addressed  at  SFR.  It  is  intended  that  the  contractor  will  identify  those  subsystems  and 
components  applicable  to  the  type  of  system  being  developed  and  the  appropriate  criteria  for  each 
subsystem  and  component  necessary  to  effectively  evaluate  and  assess  the  proposed  system  concept  and 
technical,  cost,  and  schedule  parameters,  e.g. : 

=>  For  Electrical  Power  Systems 

•  Electrical  Power  Distribution  System  (EPDS)  performance  requirements,  characteristics,  and 
operational  criteria  are  developed  and  baselined  for  the  selected  system  design  concept,  including 
power  budgets,  power  demand  (with  margins),  and  modes  of  operation  (frequency  and  duration) 

•  Selection  of  the  type(s)  of  power  supply  sources,  including  their  specific  technology  and  topology 
validated  for  the  selected  design  concept 

•  Derived  battery  life  requirements  (BOL  and  EOL)  and  any  other  unique  requirements  that  may 
impact  battery  detailed  design  baselined  for  the  selected  design  concept 

•  Battery  cell  technology(s)  selection(s)  provided  for  the  selected  design  concept;  battery 
architecture(s)  (including  hierarchy  and  traceability  to  the  system  design  concept  performance 
criteria) 

=^>  For  Software 

•  Software  architecture  to  SWCI  level  is  defined  for  the  selected  system  design  concept 

•  SWCI  external  and  internal  interfaces  are  defined 

•  Software-related  segment  requirements  allocation  to  SWCIs  is  defined 

•  Processing  capacity  and  throughput  requirements  are  validated  for  the  selected  design  concept 
solution 

•  Reprogrammability  criteria  and  capability  are  correlated  with  and  validated  for  the  selected 
system  design  concept 

•  An  estimate  of  ESLOC  to  reflect  software  architecture  and  hierarchy  of  the  selected  design 
concept  is  developed  and  demonstrates  traceability  to  the  design  concept’s  software  performance 
requirements 

•  With  the  preliminary  software  maintenance  plan  complete,  software  maintenance  supports  the 
program  logistic  plan 
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•  Software  risk  management  plan  is  complete,  software  risk  management  tool  shall  be  compatible 
with  program  risk  management  tools 

•  Software  risk  management  and  mitigation  processes  are  validated  for  the  selected  system  design 
concept 

20.4.3  System,  Segment,  and  Subsystem  Verification  and  Validation 

Evidence  of  System,  Segments,  and  Subsystem  Design  Concepts  V&V  requirements  maturity  criteria 
examples  at  SFR: 

A.  System  V&V  strategies,  concepts,  and  methodologies  are  validated  for  the  proposed  design  concept 
solution  with  acceptance  rationale: 

1.  Design  concept  strategies  established  to  demonstrate  and  verify  major  system  performance 
requirements  and  parameters,  e.g.: 

a.  V&V  strategy  and  methodology  addresses  system,  segment,  and  subsystem  and  component- 
level  verification  approaches  for  the  selected  design  concept 

b.  V&V  strategy  and  methodology  addresses  analytical,  modeling  and  simulation,  and  testing 
strategies  and  techniques  for  the  selected  design  concept,  e.g.: 

(1)  Analytical,  modeling  and  simulation  (M&S)  and  testing 

(2)  Use  of  new  technology  qualification  practices,  system(  s)-level  demonstrations  and  tests 

(3)  External  organizations  and/or  facilities,  and  resource  requirements  for  the  selected  design 
concept  and  support 

(4)  Uses  of  proven  practices  are  defined  with  references 

2.  Updated  system  VCRM  is  complete  and  consistent  with  system  requirements  and  external 
interface  requirements,  e.g.: 

a.  Segment  and  subsystem  VCRMs  are  complete,  consistent  with  segment  and  subsystem 

b.  Requirements  are  traceable  to  the  system  VCRM 

c.  The  updated  system  VCRM  and  the  segment  and  subsystem  VCRMs  are  baselined  and  under 
configuration  management 

d.  Verification  methods  in  the  system  and  segment  and  subsystem  VCRMs  are  adequate  to 
verify  system,  segments,  and  subsystems 

B.  System,  Segment,  and  Subsystem  operational  functions  and  environments  for  the  selected  design 
concept  are  identified  are  defined  and  are  traceable  to  the  contractor’s  operations  concept  and  the 
Functional  Baseline: 

1.  System  V&V  test  environments  are  defined  and  traceable  to  the  system  performance  specification 
for  the  selected  design  concept 

2.  Demonstrate  that  environmental  parameters  are  correlated  with  verification  strategies  and 
methodology  for  the  selected  design  concept 

C.  DT&E  elements  are  defined  for  the  selected  system  design  concept  and  execution  strategy  developed 

D.  OT&E  requirements  analyses  are  completed  and  test  criteria  defined  (in  conjunction  with  AF 
Operations  Test  and  Evaluation  Center  (AFOTEC))  for  the  selected  system  design  concept  traceable 
to  operational  T&E  trade  study  results: 

1 .  Analysis  results  are  traceable  to  inputs  and  requirements  from  all  potential  stakeholders 

2.  V&V  test  requirements  are  derived  and  integrated  into  program  planning  and  design  concept 

3.  Resource  and  programmatic  requirements  and  issues  are  identified  that  may  impact  program 
technical,  cost,  or  schedule  parameters,  e.g.: 
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a.  Requirements  and  architecture(s)  of  test  beds  and  test  facilities  are  documented  and  proven 
suitable  for  the  selected  design  concept’s  system,  segment,  and  subsystem,  and  interface 
requirements  verification  (V&V) 

b.  For  critical  hardware  and  software  items,  V&V  resources  (e.g.,  simulators,  test  beds,  test 
facilities)  are  identified  and  plans  and  schedules  are  in  place  for  their  development  or 
procurement 

E.  Test  requirements  and  test  data  collected  to  date  for  the  selected  design  concept  are  traceable  to 
operational  requirements  via  specifications  defined  by  the  V&V  cross-reference  matrixes  (VCRMs), 
e.g.,  use  of  comparative  test  data  to  anchor  representative  system  models  and  simulations  to  real- 
world  environments  and  system  functional  performance  requirements  is  demonstrated 

F.  V&V  risk  areas  and  mitigation  strategies  are  baselined  for  the  selected  design  concept: 

1.  V&V  test  deficiencies,  including  those  based  on  technology  deficiencies  are  identified  and 
characterized  for  the  selected  design  concept 

2.  Risk  mitigation  strategies  are  developed  and  integrated  into  the  system  risk  model,  including 
resource  requirements  for  the  selected  design  concept 

20.4,4  Engineering  Disciplines  and  Specialty  Engineering 

Evidence  of  Engineering  Discipline  and  Specialty  Engineering  identification  and  assessment  maturity 
criteria  examples  (categories  listed  in  A  through  R  below)  at  SFR  in  terms  of,  e.g.: 

1 .  Key  performance  requirements 

2.  Key  performance  parameters 

3.  Use  of  heritage  systems,  components,  and  technology 

4.  Use  of  new  designs 

A.  Parts,  Materials,  and  Processes  (PM&P) 

1.  PM&P  functional  requirements  are  validated  for  the  selected  design  concept 

2.  Assessment  of  environments  and  environmental  parameters  impacting  parts  performance  for  the 
selected  system  design  concept  is  completed 

3.  Parts  engineering  design  strategy  for  the  proposed  design  solution  is  selected,  including  risk 
assessments,  technologies,  sources  of  supply,  and  the  common  quality  levels  (i.e.,  reliability)  of 
the  parts 

B.  Test  and  Evaluation  (T&E) 

1.  T&E  strategy  is  initially  developed,  correlated  with  the  selected  design  solution  concept 
illustrating  all  test  objectives,  test  environments,  and  test  resources  to  ensure  compliance  with 
design  and  specified  requirements 

2.  T&E  methodology(s)  is  correlated  with  the  selected  design  concept,  e.g.,  test  methodology(s) 
outlines  all  test  approaches  for  the  system  components  critical  to  verifying  system  technical 
requirements  and  is  tailored  to  the  characteristics,  effectivity(s),  and  margins  of  each  particular 
test  item 

3.  Test  and  verification  methodology  for  data  gathering,  reduction,  and  analysis  for  the  selected 
system  design  concept  is  validated,  including  test  environment(s),  operations,  and  procedures  to 
be  performed,  data  acquisition  requirements,  documentation,  methods  of  analysis,  and  pass-fail 
(i.e.,  success)  criteria,  e.g.: 

a.  Evaluation  of  integration  and  verification  test  planning 

b.  Integration  and  test  plan  from  unit  to  system  level  for  dynamics  environment  testing 

c.  Identification  of  development  tests  necessary  to  evaluate  system  and  subsystem  requirements 
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d.  Assurance  that  verification  planning  is  adequate  at  system  and  segment  levels 

e.  Selection  of  test  facilities 

C.  Survivability  and  Vulnerability 

1 .  Survivability  and  vulnerability  threat  assessments  for  the  selected  design  concept  and  KPPs  are 
validated  for  each  assessed  threat  defining  the  categories  of  expected  threats,  threat  environments, 
and  their  likelihood  of  occurrence 

2.  System  and  threat  interaction  analyses  are  performed  for  the  selected  system  design  concept  to 
establish  allowable  margins  for  each  threat 

3.  Survivability  design  criteria  are  derived  from  threat  analyses  validated  to  support  the  selected 
design  concept  solution  to  mitigate  each  assessed  threat 

D.  Environmental  Safety  and  Occupational  Health  (ES&OH) 

1.  Life  cycle  environments  for  the  selected  design  concept  are  fully  defined  with  rationale  and  are 
traceable  to  the  system  requirements  baseline  and  performance  criteria 

2.  Data  compiled  to  complete  Programmatic  Environmental,  Safety  and  Occupational  Health 
Evaluation  (PESHE)  compliance  objectives  is  validated  for  the  selected  system  design  concept 

3.  Hazardous  materials  management  and  pollution  prevention  tasks  are  identified  and  prioritized  for 
the  selected  system  design  concept 

4.  Environmental  and  health  evaluations  of  hazardous  materials  have  been  conducted 

E.  Mass  Properties 

1.  A  mass  properties  budget  is  validated  for  the  selected  system  design  concept,  including  mass 
properties  growth  allocations  and  metrics 

2.  Parameters  for  weight  growth,  center  of  gravity,  and  moments  of  inertia  predictions  are  validated 
for  the  selected  system  design  concept 

F.  System  Security  Engineering  (SSE),  Information  Assurance  (IA),  Communications  Security 

(COMSEC),  and  Program  Protection  (PP)  for  (SFR): 

1.  Requirements  implementation  into  the  selected  design  concept  IAW  DoD  and  AF  policies, 
directives,  and  system  specifications  are  verified 

2.  Design  implementation  for  program  protection  measures  includes  integration  within  the  selected 
system  design  concepts  and  includes  SSE,  1A,  and  COMSEC  requirements 

3.  System  security  approaches  includes  an  acquisition  team,  a  security  specification,  and  updates  of 
security  concept,  threat,  vulnerability  and  risk  assessments,  protection  countermeasures,  and 
security  test  and  evaluation  requirements 

4.  Information  Assurance  and  COMSEC  approaches,  to  include  certification  and  accreditation  using 
the  DIACAP,  sufficiently  address  system  and  data  protection,  availability,  integrity, 
confidentiality,  authentication,  and  nonrepudiation  for  the  selected  design 

5.  Program  baseline  cost  estimates  for  SSE,  COMSEC,  Program  Protection,  and  Information 
Assurance  implementation  and  sustainment  are  refined 

G.  Interoperability 

1.  DoD  Information  Standards  Repository  (DISR)  standards  selected  for  the  selected  design  concept 
are  shown  to  meet  the  system  and  mission  interoperability  requirements  (i.e.,  must  be  DISR 
compatible  and  compliant) 

2.  New  and  unique  standards  outside  DISR  recommended  for  the  selected  design  concept  are 
submitted  for  approval  and  incorporation  into  DISR  (i.e.,  new  data  formats,  data  exchange 
protocols  and  schemas,  Ethernet  alternatives) 
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3.  Interoperability  architecture  requirements  are  defined  for  the  selected  system  design  concept 

4.  Interoperability  analyses  for  the  selected  design  concept  are  completed  and  results  shown  to 
ensure  compatibility  and  define  interrelationships  between  users  and  operators 

H.  Reliability,  Dependability,  and  Maintainability  (RD&M) 

1.  R&M  requirements  and  characteristics  are  correlated  with  and  validated  against  requirements  for 
the  selected  design  concept  (i.e.,  Mean  Mission  Duration  (MMD),  Availability  (Ao)  and 
Dependability  (Do),  MTBF,  MTTR,  failure  modes,  single  point  of  failure,  redundancy,  etc.) 

2.  R&M  analyses  are  completed  and  results  fed  into  overall  system  architecture  for  the  selected 
design  concept,  e.g.: 

a.  Methodologies  for  defining  environmental  and  thermal  stress  screening  (ESS  and  TSS)  are 
baselined  and  validated  for  the  selected  design  concept 

b.  Packaging,  handling,  storage,  and  transportability  (PHS&T)  environmental  requirements  are 
baselined  and  incorporated  into  the  R&M  program  for  the  selected  design  concept 

3.  A  functional  FMECA  is  provided  for  the  update  system  architecture,  e.g.,  identify  the 
measurement  parameters  and  anomalous  limits  for  supporting  acceptance  test  and  integration  test 
and  ADR 

I.  EMI  and  EMC 

1.  Electromagnetic  interference  control  approaches  are  correlated  with  and  validated  for  the  selected 
design  concept 

2.  Internal  and  external  EMI  and  EMC  requirements  are  baselined  and  validated  for  the  selected 
design  concept 

3.  EMI  susceptibility  requirements  and  constraints  are  correlated  with  and  validated  for  the  selected 
design  concept  (e.g.,  passive  modulation,  transmitter  RFI  with  vehicle  receivers  and  ordnance, 
radiated  effects  on  power  buses,  lightning  and  surge  protection) 

4.  EMI  and  EMC  critical  environmental  characteristics  and  sensitive  elements  are  baselined  and 
validated  for  the  selected  design  concept 

J.  Human  Systems  Integration  (HSI) 

1.  User  interface  hardware  and  software  requirements  for  operators,  users,  maintainers,  and 
sustainers  are  allocated  to  the  selected  design  concept 

2.  Usability,  maintainability,  operability,  and/or  supportability  requirements  are  decomposed  from 
system  functional  requirements  and  allocated  to  the  selected  design  concept 

3.  Operational  manning,  workload,  and  skill- level  requirements  are  allocated  for  the  selected  design 
concept 

4.  All  HSI-related  requirements,  standards,  and  standard  practices  are  allocated  to  all  subordinate 
contracting  activities  for  the  selected  design  concept 

5.  HSI  standards  for  predetermined  requirements  are  incorporated  into  the  selected  design  concept 

K.  Manufacturing  and  Producibility 

1 .  Manufacturing  and  producibility  engineering  studies  are  completed;  application  of  derived  results 
are  shown  to  satisfy  the  selected  design  concept,  e.g.,  one  best  method,  operation  instructions, 
assembly  aids,  jigs  and  fixture  designs,  short  interval  schedules,  factory  and  bench  layout 

2.  Producibility  trade  studies  for  the  selected  design  concept  demonstrate  that  the  manufacturing 
processes  chosen  satisfy  the  design  concept 

3.  Producibility  analysis  results  derived  for  the  selected  design  concept  indicate  a  cost-effective, 
producible,  and  testable  product  design 
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L.  Life  Cycle  Logistics 

1 .  Supportability  requirements  and  design  factors  are  validated  for  the  selected  design  concept 

2.  System-level  design  factors  for  the  selected  design  concept  are  verified  for  the  following  logistics 
elements:  design  interface,  supply  support,  test  equipment,  manpower  and  personnel,  training  and 
training  equipment,  PHS&T,  facilities,  computer  resources,  technical  data,  and  maintenance 
planning 

3.  Logistics  management  information  (LMI)  is  completed  and  validated  in  support  of  the  FBL  for 
the  selected  design  concept  and  includes  all  updated  supportability  trade  studies  and  analysis 
results 

M.  System  Safety 

1.  System  safety  requirements  are  validated  for  the  selected  system  design  concept,  including 
refinement  of  system  safety  risk  analysis  results  and  reassessment  of  mitigation  approaches 

2.  System  hazard  analysis  is  completed  and  a  balanced  list  of  prioritized  safety  hazards  is 
established  for  the  test,  operation,  and  disposal  of  the  selected  design  concept,  e.g.: 

a.  Initial  Hazardous  Materials  Management  Plan  (HMMP)  to  eliminate,  minimize,  or  control 
hazardous  materials  for  the  selected  design  concept 

b.  Critical  human  safety  and  health  factors  are  identified  and  validated  for  the  selected  system 
design  concept  incorporated  into  the  system  safety  program  architecture 

3.  A  baselined  hazardous  materials  list  affecting  the  selected  system  design  concept  is  compiled  and 
prioritized  based  on  NEPA  and  OSHA  criteria,  i.e.,  PELs,  toxicity,  volatility,  transportability 

4.  Demilitarization  and  disposal  considerations  for  the  hazardous  materials  is  considered 

N.  Contamination  Control 

1.  Contamination  control  needs  and  approaches  (i.e.,  normal,  medium,  or  challenging  and  stressing 
contamination  control)  are  validated  for  the  selected  design  concept 

2.  Survey  of  materials  is  conducted  and  results  evaluated  to  identify,  validate,  and  prioritize 
outgassing  properties  for  the  selected  design  concept 

O.  Quality  Assurance 

1.  Quality  and  product  assurance  requirements  are  correlated  with  and  validated  for  the  selected 
design  concept 

2.  Verification,  inspection,  and  test  approaches  are  validated  for  the  selected  design  concept 

P.  Environmental  Considerations 

1.  Environmental  studies  are  completed  for  the  selected  design  concept,  traceable  to  the  system 
architecture  and  R&M  requirements 

2.  A  robust  test  program  is  defined  for  environmental  effects  on  design,  e.g.: 

a.  Thermal  test  and  evaluation  strategies  are  developed  and  validated  for  the  selected  design 
concept 

b.  Reliability  thermal  trade  analyses  are  completed  and  made  traceable  to  system  architecture 
and  R&M  requirements 

Q.  Software 

1.  Software  system  architecture  and  design  (including  interfaces)  are  completed  and  validated  for 
the  selected  design  concept  to  the  level  of  detail  consistent  with  the  software  development  life 
cycle  being  used,  including,  e.g.: 

a.  Software  architecture  and  design  satisfy  design  requirements 
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b.  Software  requirements,  architecture  and  design  have  bidirectional  traceability  to  each  other 

c.  Software  requirements,  architecture  and  design  have  bidirectional  traceability  to  higher  level 
requirements,  architecture  and  design 

d.  Software  requirements  for  operations,  maintenance,  and  training  needs 

e.  Software  requirements  for  dependability,  reliability,  maintainability,  and  availability 

f.  Software  requirements  for  supportability  applicable  to  both  software  and  hardware 

g.  Software  safety  requirements  applicable  to  both  software  and  hardware 

h.  Software  requirements  for  information  assurance 

i.  Software  requirements  for  human  systems  integration  (HSI) 

j.  Software  requirements  for  interoperability  with  external  elements  referencing  all  appropriate 
interoperability  and  open  system  standards 

k.  Software  requirements  for  margins  (e.g.,  memory  and  storage  capacity,  processor  throughput, 
and  communications  bandwidth) 

l.  Use  of  open  systems  standards  that  satisfy  all  applicable  interoperability-related  requirements 

m.  Software  requirements  allocation  to  COTS  (commercial  items),  GOTS,  and  reuse  products 

n.  Non- developmental  items  (NDI)  (e.g.,  COTS  (commercial  item),  GOTS,  and  reuse  software) 
have  been  fully  integrated  into  the  components  of  the  system  and  software  architectures 

o.  Non- developmental  items  (NDI)  (e.g.,  COTS,  GOTS,  and  reuse  software)  will  enable  the 
system,  segment,  and  interface  requirements  to  be  met 

2.  Software  architecture  and  design  satisfy  requirements  for  states  and  modes  as  allocated  from  the 

system  requirements 

3.  Operational  Concepts,  e.g.: 

a.  Software  architecture  and  design  satisfy  system  and  software  operational  concepts,  including 
both  nominal  and  off-nominal  scenarios  from  a  software  perspective,  e.g.,  processor  failover, 
redundancy  management 

b.  Software  architecture  and  design  satisfy  system  and  software  requirements  for  operations, 
maintenance,  and  training  needs 

c.  Software  architecture  and  design  satisfy  system  and  software  operational  concepts,  including 
operations  and  maintenance  staffing,  e.g.,  numbers,  skills,  roles,  and  positions  from  a 
software  perspective 

d.  Software  architecture  and  design  satisfy  requirements  for  supportability  and  apply  to  both 
software  and  hardware 

4.  Analysis  of  software  metrics  and  technical  performance  measures  shows  that 

a.  They  are  sufficient  to  meet  the  needs  for  program  and  engineering  management 

b.  Satisfactory  progress  has  been  made  to  date  and  indicates  continued  satisfactory  progress  in 
the  future 

c.  The  estimates  of  utilization  for  all  computer  resources,  e.g.,  processors,  memory,  storage,  and 
input  and  output  channels,  buses,  and  networks  are  within  predicted  values 

d.  Databases  and  tools  are  selected  for  metrics  and  TPM  tracking;  trending  and  reporting  are 
performing  as  planned 
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R.  Data  Storage  (Security,  Access,  Distribution,  and  Delivery) 

1.  Storage  System  Capability,  Flexibility,  and  Scalability 

a.  Analysis  identifies  needed  reliability,  maintainability,  and  availability  characteristics  of 
storage  system  environments 

b.  Capacity,  flexibility,  and  extensibility  parameters  have  been  completely  identified  that 
address  system  design  life 

c.  Key  system  components  have  been  fully  identified.  Plans  for  redundancy  are  in  place  and  are 
fully  identified,  including  storage  media  hardware  and  software  capabilities  and  types 

d.  Needs  for  storage  system  management  and  performance  optimization  (including  software 
management  tools  to  provide  appropriate  partitioning  and  addressability)  identified 

2.  Design  analysis  identified  the  operational  environments  of  the  storage  system  Data  Storage 

(Security,  Access,  Distribution  and  Delivery) 

3.  Storage  System  Capability,  Flexibility,  Scalability,  e.g.: 

a.  Analysis  identifies  needed  Reliability,  Maintainability,  and  Availability  characteristics  of 
storage  system  environments 

b.  Capacity,  flexibility,  and  extensibility  parameters  have  been  completely  identified  that 
address  the  expected  life  of  the  system 

c.  Key  system  components  have  been  fully  identified.  Plans  for  redundancy  are  in  place  and 
fully  identified,  including  storage  media  hardware  and  software  capabilities  and  types 

d.  Needs  for  storage  system  management,  performance  optimization  (including  software 
management  tools  to  provide  appropriate  partitioning  and  addressability)  are  completely 
identified 

e.  Analysis  has  fully  identified  the  operational  environments  under  which  the  storage  system 
must  operate.  Identification  of  hardening  aspects  that  must  be  addressed  is  fully  described 

4.  Storage  System  Architecture,  e.g.: 

a.  The  Storage  System  Architecture  fully  addresses  elements,  including  communications  and 
processing  capacity 

b.  The  types  of  storage  system  needs  are  identified  and  fully  integrated  into  the  architecture  This 
includes  items  such  as  centralized  vs.  distributed  storage;  online,  nearline,  and  offline  needs; 
archive  (including  hierarchical  storage  management,  if  appropriate),  backup,  and  restore;  and 
data  replication 

c.  Storage  hardware  components  such  as  RAID,  Storage  Area  Networks  (SANs),  Network 
Attached  Storage  (NAS),  and  Direct  Attached  Storage  (DAS)  have  been  identified  and  fully 
integrated  into  the  architecture 

d.  Data  management  software  capabilities  have  been  identified  and  fully  integrated  into  the 
architecture.  This  includes  items  such  as  automatic  file  migration  and  transparent  file 
retrieval;  migration  between  hierarchical  levels;  and  utilities  to  report  on  media  usage,  error 
detection,  and  identification  of  media  to  be  replaced 

5.  Security,  e.g.: 

a.  The  level  of  user  integrity  (e.g.,  access  control  lists)  has  been  identified  that  enables  the 
system  requirements  to  be  met 

b.  The  level  of  encryption  needed  has  been  identified  that  enables  the  system  requirements  to  be 
met 
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c.  The  need  for  specialized  security  capabilities,  such  as  CDS,  MLS,  and  Security  Enclaves,  has 
been  identified  and  is  included  in  the  storage  system  so  as  to  ensure  that  the  system 
requirements  are  met 

6.  Data  Distribution  Methods,  e.g.: 

a.  A  complete  list  of  data  receivers  has  been  drawn  up  to  include  both  computer  and  human 
agents 

b.  The  method(s)  of  distributing  data  to  the  various  receivers  has  been  identified.  Such  method 
may  include  Subscribe  and  Publish,  Push  and  Pull,  and  global  or  restricted  Web-based  access. 

c.  The  data  distribution  methods  are  fully  integrated  into  the  storage  architecture  and  will  enable 
the  system-level  requirements  to  be  met 

7.  Functionality,  e.g.: 

a.  Analysis  has  fully  identified  the  physical  aspects  of  the  functionality  that  may  be  needed  to 
support  the  mission 

b.  The  types  of  platforms  (server  and  client)  and  operating  systems  supported  have  been  fully 
identified 

c.  The  data  connection  and  transport  protocols  (e.g.,  fiber  channel,  infmiband,  SWCI)  have  been 
fully  identified  and  integrated  into  the  system  architecture,  enabling  the  system-level 
requirements  to  be  met 

d.  Specific  reporting  (e.g.,  usage)  and  maintenance  metrics  (e.g.,  MTBF  and  MTTR)  have  been 
identified.  Preliminary  mapping  between  metrics  and  system-level  requirements  has  been 
completed 

e.  Storage  system  hardening  design  requirements  are  defined  and  understood 

8.  Storage  System  Architecture,  e.g.: 

a.  The  Storage  System  Architecture  fully  addresses  elements,  including  communications, 
processing  capacity 

b.  The  types  of  storage  system  needs  are  identified  and  fully  integrated  into  the  architecture. 
This  includes  items  such  as  centralized  vs.  distributed  storage;  online,  nearline,  and  offline 
needs;  archive  (including  hierarchical  storage  management,  if  appropriate),  backup,  and 
restore;  and  data  replication 

c.  Storage  hardware  components  such  as  RAID,  Storage  Area  Networks  (SAN),  Network 
Attached  Storage  (NAS),  and  Direct  Attached  Storage  (DAS)  have  been  identified  and  fully 
integrated  into  the  architecture 

d.  Data  management  software  capabilities  have  been  identified  and  fully  integrated  into  the 
architecture.  This  includes  items  such  as  automatic  file  migration  and  transparent  file 
retrieval;  migration  between  hierarchical  levels;  and  utilities  to  report  on  media  usage,  error 
detection,  and  identification  of  media  to  be  replaced 

9.  Security,  e.g.: 

a.  The  level  of  user  integrity  (e.g.,  access  control  lists)  has  been  identified  that  enables  the 
system  requirements  to  be  met 

b.  The  level  of  encryption  needed  has  been  identified  that  enables  the  system  requirements  to  be 
met 

c.  The  need  for  specialized  security  capabilities,  such  as  CDS,  MFS,  and  Security  Enclaves  has 
been  identified  and  is  included  in  the  storage  system  so  as  to  ensure  that  the  system 
requirements  are  met 
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10.  Data  Distribution  Methods,  e.g.: 

a.  A  complete  list  of  data  receivers  has  been  drawn  up  to  include  both  computer  and  human 
agents 

b.  The  method(s)  of  distributing  data  to  the  various  receivers  has  been  identified.  Such  methods 
may  include  Subscribe  and  Publish,  Push  and  Pull,  and  global  or  restricted  Web-based  access 

c.  The  data  distribution  methods  are  fully  integrated  into  the  storage  architecture  and  will  enable 
the  system-level  requirements  to  be  met 

11.  Functionality,  e.g.: 

a.  Analysis  has  fully  identified  the  physical  aspects  of  the  functionality  that  may  be  needed  to 
support  the  mission 

b.  The  types  of  platforms  (server  and  client)  and  operating  systems  supported  have  been  fully 
identified 

c.  The  data  connection  and  transport  protocols  (e.g.,  fiber  channel,  infmiband,  SWCI)  have  been 
fully  identified  and  integrated  into  the  system  architecture,  enabling  the  system-level 
requirements  to  be  met 

d.  Specific  reporting  (e.g.,  usage)  and  maintenance  metrics  (e.g.,  MTBF  and  MTTR)  have  been 
identified.  Preliminary  mapping  between  metrics  and  system-level  requirements  has  been 
completed 

20.4.5  Integrated  Technical  Risk  Management  and  Mitigation 

Evidence  of  technical  risk  management  (RM)  process  maturity  criteria  examples,  as  a  key  component  of 
an  integrated  program  (technical,  cost,  schedule,  and  performance)  RM  and  Mitigation  (RM&M)  process 
at  SFR: 

A.  Supporting  data  for  RM&M  can  encompass  such  items  as: 

1.  Program  schedule,  technical  and  funding  risk  assessment  ranking,  monitoring  and  documentation 

adequacy 

2.  Top  five  program-level  risks  (technical,  performance,  cost,  and  schedule) 

3.  Risk  management  database  and  tools  for  risk  metrics  collection,  analysis,  tracking,  and  reporting 

4.  Risk  mitigation  and  reduction  strategies,  e.g.: 

a.  Bum-down  plans  that  are  linked  to  dependencies  to  the  Program  IMP,  IMS,  and  WBS 

b.  Continuous  risk  monitoring  and  review,  identification,  assessment,  and  ranking 

c.  Technology  and  manufacturing  readiness  level  (TRL  and  MRL)  assessments  and  metrics 

d.  Requirements  risk  monitoring 

e.  Software  risk  management  of  critical  software  issues,  e.g.,  complexity,  size,  processing 
speed,  throughput,  schedules,  COTS  availability,  legacy  reuse  suitability,  and  software 
development  processes  and  tools 

f.  A  comprehensive  risk  assessment  for  the  follow-on  phases 

g.  TRL  and  MRL  assessments,  metrics 

h.  Thresholds  and  appropriate  action  plans  for  cases  when  thresholds  are  breached 

B.  The  risk  mitigation  strategies: 

1 .  Are  feasible,  and  alternative  courses  of  action  are  identified 

2.  Demonstrate  that  a  degree  of  maturity  exists  in  all  aspects  of  the  system,  segment,  interface,  and 

program  to  allow  the  program  to  proceed  to  PDR  with  an  acceptable  risk. 
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Appendix  C  -  Software  Requirement  and  Architecture  Review  (SAR) 


30.  Software  Requirement  and  Architecture  Review  (SAR) 

The  SAR  is  a  formal,  multidisciplinary  review  of  the  software  requirements,  architecture,  and  test 
planning  technical  products,  software  development  processes,  and  current  state  of  the  software 
development. 

The  SAR  shall  be  held  for  individual  Software  Configuration  Item  (SWCI)  or  a  collection  of  related 
SWCIs,  as  defined  by  the  contractor  and  approved  by  the  contracting  agency.  The  SAR  shall  be  held  after 
theSFR. 

30.1  General 

The  SAR  replaces  the  former  Software  Specification  Review  (SSR)  and  also  was  revised  to  reflect  the 
modem  systems’  dependence  on  complex  software  for  successful  operation  and  mission  execution.  These 
systems  involve  complex  combinations  of  hardware  and  software  with  complex  external  and  internal 
interfaces.  They  are  usually  unprecedented  (have  never  been  built  before)  and  have  high  reliability  and 
integrity  requirements.  The  size  of  the  software  in  new  systems  under  development  can  range  from  105  to 
107  source  lines  of  code  (SLOC). 

In  response  to  the  system  acquisition  strategy,  the  supplier  selects  a  software  development  life  cycle 
model.  This  can  be  a  waterfall  model,  where  the  supplier  designs,  builds,  tests,  and  delivers  the  system 
only  once,  or  an  incremental  or  evolutionary’  model,  where  the  supplier  designs,  builds,  tests,  and 
iteratively  delivers  multiple  versions  of  the  system  with  increasing  capability. 

The  positioning  of  the  SAR  in  the  software  development  life  cycle  is  dependent  upon  the  life  cycle  model 
in  use  for  the  software  under  review.  Software  is  always  developed  according  to  a  particular  life  cycle 
model.  Although  other  types  of  life  cycle  models  are  in  use,  the  most  common  types  are  the  waterfall, 
incremental,  and  evolutionary. 

In  the  waterfall  life  cycle  model  (Figure  3),  the  software  is  developed  in  a  “once  through”  fashion,  where 
the  sequence  of  software  requirements  definition,  software  architectural  and  detailed  design,  software 
implementation,  and  software  testing  (unit,  integration,  and  qualification  testing)  occurs  only  once,  as 
shown  in  Figure  3.  In  the  case  of  the  waterfall  life  cycle  model,  the  SAR  is  positioned  at  the  completion 
of  the  software  architectural  (high-level)  design,  as  shown  in  Figure  3.  When  the  software  under  review  is 
developed  using  the  waterfall  life  cycle  model,  the  SAR  shall  be  completed  before  the  system  PDR. 

In  the  incremental  and  evolutionary  life  cycle  models,  software  is  developed  in  a  series  of  builds.  A  build 
is  a  version  of  the  software  (or  system)  that  meets  a  specific  subset  of  the  requirements  that  the  completed 
software  (or  system)  will  meet.  There  are  two  common  types  of  iterative  life  cycle  models —  the 
incremental  and  the  evolutionary. 

In  the  incremental  life  cycle  model,  the  software  requirements  are  defined  first,  as  depicted  in  Figure  4. 
Then  the  software  is  developed  in  a  series  of  builds,  where  each  build  adds  to  the  previous  build  and 
enhances  its  capabilities.  In  the  incremental  life  cycle  model,  each  build  consists  of  a  once-through 
sequence  of  software  requirements  assessment,  software  architectural  and  detailed  design,  software 
implementation,  and  software  testing  (unit,  integration,  and  qualification  testing).  Thus,  while  the 
software  requirements  are  defined  up  front  in  this  life  cycle  model,  the  software  architectural  design  is 
defined  iteratively  as  the  builds  proceed.  Because  of  this,  the  SAR  is  performed  iteratively  at  the 
conclusion  of  the  software  architectural  design  for  each  incremental  build.  The  full  set  of  software 
requirements  for  the  software  under  review  is  reviewed  in  the  Build  1  SAR,  while  the  software 
architecture  and  other  information  is  reviewed  on  an  incremental  basis  as  each  build  proceeds. 
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★ 


Position  of  SAR 


Figure  3.  Waterfall  Software  Development  Life  Cycle  Model 


Figure  4.  Incremental  Software  Development  Life  Cycle  Model 

The  evolutionary  life  cycle  model  is  similar  to  the  incremental  life  cycle  model,  with  the  exception  that, 
in  the  evolutionary  life  cycle  model,  the  builds  are  based  upon  the  requirements  allocated  to  software 
from  the  parent  specification,  as  depicted  in  Figure  5.  Then  each  build  consists  of  a  once-through 
sequence  of  software  requirements  definition,  software  architectural  and  detailed  design,  software 
implementation,  and  software  testing  (unit,  integration,  and  qualification  testing).  The  distinction  between 
the  incremental  and  evolutionary  life  cycle  models  is  thus  whether  or  not  the  software  requirements  are 
defined  up  front  (incremental)  or  within  each  build  for  that  build  only  (evolutionary).  (Note  that  the 
waterfall  life  cycle  model  can  be  thought  of  as  an  evolutionary  life  cycle  model  that  has  only  one  build.). 
For  the  evolutionary  life  cycle  model,  the  SAR  is  performed  iteratively  at  the  conclusion  of  the  software 
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architectural  design  for  each  build,  and  the  software  architecture  and  other  information  is  reviewed  on  an 
iterative  basis  as  each  build  proceeds. 

For  iterative  life  cycle  models,  each  SAR  shall  review  the  technical  products  and  state  of  the  software  for 
the  current  build,  the  impact  of  the  previous  builds  upon  the  current  build,  and  the  impact  of  the  current 
build  upon  subsequent  builds.  For  iterative  life  cycle  models,  each  SAR  shall  address  the  collection  of 
SWCIs  that  integrate  together  to  form  the  builds. 


System  Requirements  Definition 


Software  Build  1 


Hardware  Requirements  Def. 


Software  Build  2 
Regression  Testing 


Preliminary  Design 


Detailed  Design 


Fabrication  &  Assembly 
Test 


Software  Build  n 
Regression  Testing 


Hardware  Integration 
Hardware  Qual.  Testing 


Each  build  consists  of 
SW  Reqs.  Definition 
SW  Architectural  Design 
SW  Detailed  Design 
SW  Implementation 
SW  Unit,  Integration  and 
Qualification  Testing  . 


HW/SW  Integration  and  Testing 
System  Qualification  Testing 


Operations  and  Maintenance 
Re-validation/Re-verification 


Position  of  SAR 


Figure  5.  Evolutionary  Software  Development  Life  Cycle  Model 

There  are  other  life  cycle  models  in  addition  to  these  three  basic  types,  and  there  are  life  cycle  models  that 
are  combinations  of  two  or  more  of  these  basic  types.  The  placement  of  the  SAR  for  these  three  life  cycle 
models  as  described  above  should  be  tailored  for  life  cycle  models  other  than  these  three  basic  patterns 
and  for  combinations  of  two  or  more  of  these  basic  patterns.  The  software  life  cycle  model(s)  in  use,  the 
positioning  of  the  Software  Requirements  and  Architecture  Review(s)  within  the  life  cycle  model(s),  and 
the  relationship  of  the  positioning  of  the  SAR(s)  with  respect  to  the  other  major  reviews  defined  in  this 
standard  (as  tailored  by  the  contract)  are  described  in  the  Software  Development  Plan  (SDP).  The 
requirements  for  the  SDP  are  specified  in  the  Software  Development  Standard  for  Space  Systems  - 
Aerospace  TOR-2004(3909)-3537,  a.k.a.  SMC-S-012. 

30.2  Objective 

The  SAR  shall  be  held  after  SFR.  The  SAR  shall  be  a  formal  review  of  an  S WCI’s  requirements  and 
architecture  as  specified  in  the  Software  Requirements  Specification  (SRS)  and  the  Interface 
Requirements  Specification(s)  (IRS(s)),  and  Software  Architecture  Description  (SAD),  IAW  SMC-S-012. 

Prior  to  and  in  preparation  for  the  SAR,  the  developer  is  to  define  and  record  the  software  requirements 
based  on  the  analysis  of  system  requirements,  the  system  design,  and  other  considerations  to  be  met  by 
each  software  item,  along  with  the  methods  and  levels  for  verifying  each  requirement,  and  the  traceability 
between  the  software  item  requirements  and  their  parent  requirements.  Traceability  shall  be  bidirectional. 
The  result  is  to  include  all  applicable  items  in  the  Software  Requirements  Specification  (SRS)  DID  and 
Interface  Requirements  Specification  (IRS)  DID,  as  defined  in  the  Software  Development  Plan  (SDP). 

A  collective  SAR  for  a  group  of  configuration  items,  treating  each  configuration  item  individually,  may 
be  held  when  such  an  approach  is  advantageous  to  the  contracting  agency.  Its  purpose  is  to  establish  the 
allocated  baseline  for  preliminary  SWCI  design  by  demonstrating  to  the  contracting  agency  the  adequacy 
of  the  SRSs,  IRSs,  and  Operational  Concept  Description  (OCD). 
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The  objectives  of  the  SAR  are  to  determine  whether: 

a.  The  software  requirements  and  architectural  design  are  adequate  for  meeting  the  higher-level 
requirements  allocated  to  software 

b.  The  software  requirements  and  architectural  design  are  sufficiently  mature  to  proceed  with 
dependent  software  and  system  development  activities 

c.  The  software  processes  are  sufficiently  defined,  mature,  and  effective  for  developing  the  software 
needed  to  meet  system  requirements  and  operational  needs,  and  are  suitable  for  the  program 
scope  and  complexity 

d.  The  software  test  plans  are  sufficiently  robust  to  ensure  thorough  testing  of  the  software  products 
to  demonstrate  that  the  software  requirements  are  verified  in  the  target  environment 

e.  The  software  development  and  test  environments  are  established  and  have  adequate  capability 
and  capacity  to  meet  the  software  development  and  test  requirements  and  schedules 

f.  The  software  development  risk  is  manageable 

g.  The  software  development  costs  and  schedules  are  consistent  with  program  costs  and  schedules. 

30.3  Items  To  Be  Reviewed 

The  contractor  is  to  present  the  following  items  for  review  by  the  contracting  agency  for  the  software 
under  review: 

A.  Requirements 

1 .  Higher-level  (parent)  requirements  allocated  to  software 

2.  Software  requirements  and  software  interface  requirements 

3.  Software -related  external  and  intersegment  or  element  interface  requirements 

4.  ICDs  with  software-to-software  and  software-to-hardware  interface  requirements 

B.  Operational  Concepts 

1.  Software  operational  concepts 

C.  Software  Architectural  Design 

1.  Software  architectural  design  description 

2.  Top-level  computer  system  hardware-software  architectural  design  description 

D.  Engineering  Analyses 

1.  Software  engineering  analyses,  trade  studies,  modeling  and  simulation  results 

2.  Hardware-to-software  engineering  analyses,  hardware  vs.  software  trade  studies,  modeling  and 
simulation  results 

E.  Integration  and  Verification 

1.  Software  master  build  plan  (allocation  of  requirements,  functionality,  and  architectural 
components  to  builds) 

2.  Software  qualification  test  plans 

F.  Traceability 

1 .  Bidirectional  traceability  between: 

a.  Higher-level  requirements  (including  interface  requirements)  allocated  to  software  and 
software  requirements  (including  software  interface  requirements) 

b.  Traceability  between  software  requirements  (including  software  interface  requirements)  and 
software  architecture  components 
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c.  Traceability  between  software  requirements  (including  software  interface  requirements)  and 
software  qualification  tests 

d.  Traceability  between  requirements  and  verification  methods  and  verification  integration 
levels 

e.  Traceability  between  builds  and  software  requirements,  software  architectural  components, 
and  software  qualification  tests 

G.  Risk  Management 

1.  Software  risk  information,  including  identification,  prioritization,  and  risk-handling  plans 
(mitigation  or  other  techniques)  and  status 

H.  Costs  and  Schedules 

1.  Software  size,  effort,  cost,  schedule,  and  staffing  estimates 

2.  Software  Resources  Data  Reporting:  Initial  Developer  Report  and  Data  Dictionary  (SRDR-I) 

3.  Software  Resources  Data  Reporting:  Final  Developer  Report  and  Data  Dictionary  (SRDR-F)  for 
any  builds  completed 

4.  Software  schedules 

5.  Higher-level  schedules,  including  the  IMS 

6.  Updates  to  the  life  cycle  cost  (LCC)  and  cost  as  an  independent  variable  (CAIV)  studies 
presented  at  the  SRR  in  support  of  each  software  architecture  concept,  e.g.: 

a.  LCC  and  CAIV  modeling  and  analyses  are  applied  and  correlated  with  each  software 
architecture  concept,  e.g.,  cost  models  depicting  projected  program  development,  operational 
and  sustainment  costs  completed,  as  well  as  projected  cost  impacts  to  other  “external” 
systems 

b.  LCC  and  CAIV  methodology  is  presented  that  demonstrates  that  valid  trade  studies  were 
conducted 

I.  Engineering  and  Management  Plans 

1 .  Software  Development  Plan  ( SDP) 

2.  Other  software-related  program  plans  (e.g.,  Systems  Engineering  Management  Plan,  Risk 
Management  Plan,  Integrated  Master  Plan,  Configuration  Management  Plan,  Quality  Assurance 
Plan) 

3.  System  (hardware  and  software)  specialty  engineering  plans  (e.g.,  reliability,  safety, 
supportability,  security  (information  assurance),  and  human  systems  integration  plans) 

4.  Plans  and  status  of  the  software  engineering  environment 

5.  Plans  and  status  of  the  software  test  environments,  including  test  beds,  test  facilities,  hardware, 
software,  simulators,  and  other  testing  tools 

J.  Metrics  and  Technical  Performance  Measures 

1.  Software  metrics  reports 

2.  Software-related  TPM  reports 

3.  Software  problem  and  deficiency  report  status 
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30.4  SAR  “Acceptance  Criteria” 

At  SAR,  all  major  program  elements  and  risk  drivers  of  the  Systems  Engineering  Management  activities 
are  to  be  considered.  A  prime  expectation  of  the  SAR  is  that  the  review  will  result  in  an  approved 
software  requirement  and  architecture  baseline  that  can  be  brought  under  change  control  and  a 
determination  that  the  approved  baseline  can  be  implemented  within  constraints  of  the  cost,  schedule,  and 
performance  requirements.  In  preparation  for  and  scheduling  of  the  SAR,  the  contractor  shall  demonstrate 
the  following  minimum,  but  not  all-inclusive,  list  of  “Acceptance  Criteria,”  as  specifically  tailored  by 
contract,  along  with  all  applicable  engineering  activities,  to  the  satisfaction  of  the  contracting  agency: 

A.  Requirements 

1 .  The  higher- level  requirements  allocated  to  software  are  complete  and  stable 

2.  Software  requirements  (including  software  interface  requirements)  have  been  specified  to  the 
level  of  completeness  called  for  in  the  software  development  plan  based  on  the  selected  software 
life  cycle  model 

3.  Software  requirements  (including  software  interface  requirements)  are  correct,  complete, 
consistent,  feasible,  verifiable,  and  clearly  and  unambiguously  stated 

4.  Software  requirements  (including  software  interface  requirements)  are  traced  to  and  fully 
implement  their  parent  requirements 

5.  Software  requirements  include  necessary  requirements  derived  from  the  system  and  software 
architecture,  system  operational  concepts,  trade  studies,  or  design  decisions 

6.  Each  software  requirement,  including  software  interface  requirements,  has  one  or  more  valid 
verification  methods  and  verification  levels  specified,  and  those  methods  and  levels  are  sufficient 
to  fully  verify  the  requirement 

B.  Operational  Concepts 

1.  Software  operational  concepts  include  nominal  and  off-nominal  scenarios  from  a  software 
perspective  (e.g.,  start-up,  initialization,  shutdown,  processor  failover,  redundancy  management, 
recovery  and  restoral)  consistent  with  the  system  and  software  architectures 

2.  Software  operational  concepts  include  information  exchange  with  external  interfacing  systems 

3.  Software  operational  concepts  include  scenarios  for  operational  workloads 

4.  Software  operation  concepts  are  consistent  with  system  operational  concepts 

C.  Architecture  and  Design 

1.  The  software  architecture  has  been  defined  to  the  level  of  completeness  called  for  in  the  software 
development  plan,  based  on  the  selected  software  life  cycle  model 

2.  The  software  architecture  views,  including  the  physical,  logical,  developmental,  process,  and 
behavioral  (user)  views,  are  correct,  complete,  consistent,  clear,  and  unambiguous 

3.  Nondevelopmental  items  (NDI)  (e.g.,  COTS,  GOTS,  and  reuse  software)  have  been  fully 
integrated  into  the  components  of  the  software  architecture 

4.  The  software  architecture,  including  the  nondevelopmental  items  (NDI)  (e.g.,  COTS,  GOTS,  and 
reuse  software),  will  enable  the  higher-level  requirements  allocated  to  software,  the  software 
requirements,  and  the  software  interface  requirements  to  be  met 

5.  The  design  of  each  software  item  has  been  elaborated  to  the  level  of  software  units,  consistent 
with  the  software  development  plan  and  the  selected  software  life  cycle  model 

6.  The  design  of  each  software  item  is  clear,  correct,  complete,  consistent,  and  unambiguous,  and 
adequately  addresses  the  following: 

a.  High-level  design  of  all  external  and  internal  interfaces 
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b.  High-level  design  of  all  files,  databases,  shared  memory,  etc.,  and  their  storage  and  access 
methods 

c.  High-level  design  of  user  interface  screens  and  human  and  system  interactions 

d.  Source  for  each  unit  of  the  software  item  (i.e.,  COTS,  unmodified  reuse,  modified  reuse,  or 
newly  developed  code)  and  programming  language(s)  to  be  used 

e.  Selected  COTS  software  products  and  installation  and  configuration  design  decisions 

7.  The  high-level  design  of  each  software  item  properly  implements  all  applicable  standards  (e.g., 
interface  standards,  graphical  user  interface  (GUI)  standards) 

8.  The  software  architecture  adequately  addresses  use  of  open  systems  standards  and  satisfies  all 
applicable  interoperability-related  requirements 

9.  The  software  architecture  adequately  addresses  end-to-end  processing  (including  timelines  and 
capacity) 

10.  The  software  architecture  adequately  addresses  operational  database  management  and  control 

11.  Computing  resources  (e.g.,  processors,  cache,  memory,  buses,  and  networks)  are  selected  and 
appropriately  incoiporated  into  the  top-level  computer  system  hardware  and  software 
architecture,  and  will  enable  higher-level  requirements  allocated  to  software,  the  software 
requirements,  and  the  software  interface  requirements  to  be  met 

12.  The  software  architecture  meets  appropriate  functional  and  performance  requirements  for  each 
state  and  mode 

13.  The  architecture  adequately  addresses  supportability,  including  integrated  hardware-software 
diagnostics,  fault  detection,  isolation,  localization,  restoral,  and  repair 

14.  The  software  architecture  adequately  addresses  reliability,  maintainability,  and  availability 
requirements  allocated  to  the  computer  hardware  and  software  subsystems 

D.  Engineering  Analysis 

1.  Engineering  analyses,  models,  and  simulations  adequately  demonstrate  that  the  software 
architecture,  together  with  the  computer  resources  (hardware  and  software)  that  have  been 
selected,  will  meet  the  Key  Performance  Parameters  (KPPs)  and  driving  requirements 

2.  Reliability,  maintainability,  and  availability  analyses  are  consistent  with  the  software  architecture 
and  with  the  computer  resources  (hardware  and  software)  that  have  been  selected,  and 
appropriately  include  the  contribution  of  software 

3.  Safety,  information  assurance,  and  human  systems  integration  analyses  are  consistent  with  the 
software  architecture  and  with  the  computer  resources  (hardware  and  software)  that  have  been 
selected,  and  appropriately  include  the  contribution  of  software 

4.  Engineering  analyses  and  trade  studies  adequately  support  software  architectural  design  decisions 
about  NDI  (reuse,  COTS,  and  GOTS  software  components),  and  appropriately  consider  the 
underlying,  supporting  computer  resources  (hardware  and  software)  that  have  been  selected 

5.  Human  systems  integration  engineering  analyses  and  trade  studies  (e.g.,  operability,  operator 
workload  analysis)  demonstrate  the  adequacy  of  the  software  architecture  and  the  computer 
resources  (hardware  and  software)  that  have  been  selected,  for  the  operators  to  perform  their 
required  roles  within  the  required  timelines 

6.  Preliminary  performance  analysis  demonstrates  that  the  software  architecture,  together  with  the 
computer  resources  (hardware  and  software)  that  have  been  selected,  meet  performance 
requirements  with  adequate  margins  for  this  point  in  the  life  cycle 

7.  Engineering  analyses  and  trade  studies  demonstrate  the  adequacy  of  the  software  architecture, 
together  with  the  computer  resources  (hardware  and  software)  that  have  been  selected,  for 
meeting  the  computer  resource  margin  requirements 
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8.  All  the  above  analyses  take  into  account  actual  performance  of  existing  software  (e.g.,  prototypes, 
earlier  builds,  NDI)  on  the  selected  hardware 

9.  Engineering  models  and  simulations  have  been  used  to  demonstrate  the  adequacy  of  the 
algorithms  to  be  implemented  in  software 

E.  Integration  and  Verification 

1.  Software  qualification  test  plans  have  been  defined  to  the  level  of  completeness  called  for  in  the 
Software  Development  Plan,  based  on  the  selected  software  life  cycle  model 

2.  Software  qualification  test  plans  are  valid,  complete,  stable,  consistent  with  the  software 
architectures  and  with  higher-level  test  plans,  and  consistent  with  the  qualification  requirements 
for  test  methods  and  test  levels  for  the  software  requirements  and  software  interface  requirements 

3.  Software  requirements  are  fully  allocated  to  the  tests  described  in  the  software  qualification  test 
plans  where  they  will  be  verified 

4.  The  master  software  build  plan  is  complete,  feasible,  executable,  and  consistent  with  the  software 
requirements,  software  architecture,  software  qualification  test  plans,  and  higher-level  schedules 

F.  Traceability 

1.  All  traceability  information  is  correct,  bidirectional,  and  consistent  with  the  higher-level 
requirements  allocated  to  software,  software  requirements,  software  interface  requirements, 
software  architectural  components,  and  software  qualification  test  plans 

2.  All  traceability  information  is  defined  to  the  level  of  completeness  defined  in  the  Software 
Development  Plan,  based  on  the  selected  life  cycle  model 

G.  Risk  Management 

1.  The  software  risk  assessment  includes  the  following  software  risks  as  appropriate: 

a.  Risks  related  to  software  size  and  complexity 

b.  Risks  related  to  requirements  allocated  to  software 

c.  Risks  related  to  the  software  aspects  of  the  system  and  software  architectures 

d.  Risks  related  to  selection  and  use  of  NDI  (COTS,  reuse,  GOTS) 

e.  Risks  related  to  selection  and  use  of  computing  resources  (e.g.,  processors,  cache,  memory, 
buses,  and  networks) 

f.  Risks  related  to  growth  margins  for  computing  resources 

g.  Risks  related  to  software  schedules 

h.  Risks  related  to  software  development,  integration,  and  verification  processes  and  tools 

i.  Risks  related  to  population,  update,  control,  and  validation  of  databases 

j.  Risks  related  to  software  and  computer  hardware  technology 

2.  A  sound  software  risk  management  plan  is  part  of  the  Software  Development  Plan  and  is 
integrated  with  program  Risk  Management  Plan 

3.  An  effective  program  risk  management  process,  including  the  software  risk  management  process, 
has  been  demonstrated  to  be  functioning 

4.  Effective  software  risk-handling  plans  are  in  place,  and  risk-handling  activities  are  being 
performed  in  accordance  with  the  plans 

H.  Costs  and  Schedules 

1.  Software  cost  models  have  been  calibrated  with  actual  data  (both  from  the  current  project  as  well 
as  past  history)  and  used  to  update  software  cost  and  schedule  estimates 
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2.  Realistic  software  cost  drivers,  such  as  complexity  and  other  parameters,  and  assumptions  are 
documented,  validated  with  documented  project  data,  and  used  in  software  cost  models  to 
develop  updated  cost  and  schedule  estimates 

3.  Software  size  estimates  are  supportable,  based  on  history,  and  consistent  with  the  software  and 
interface  requirements  and  software  architecture 

4.  Software  cost  and  schedule  estimates  have  enough  margins  to  cover  the  estimation  risk 
appropriate  to  this  point  in  time 

5.  Software  schedules  are  consistent  with  higher- level  schedules,  including  the  IMS 

I.  Engineering  and  Management  Plans 

1.  The  SDP  is  consistent  with  the  IMP,  systems  engineering  management  plan,  and  other 
management  and  engineering  plans 

2.  The  SDP  addresses  the  full  software  development  life  cycle 

3.  The  SDP  describes  an  integrated  set  of  effective  processes,  methodologies,  tools,  and 
environments  that  cover  all  software  team  members,  are  suitable  for  the  domain,  and  are 
appropriate  for  program  scope  and  complexity 

4.  The  SDP  describes  selected  software  development  life  cycle  models  that  are  feasible,  appropriate 
for  program  scope  and  complexity,  and  used  consistently  by  all  team  members 

5.  Software  processes,  standards,  procedures,  and  conventions  for  use  throughout  the  life  cycle  are 
documented,  validated,  and  included  with  the  SDP 

6.  The  existing  and  planned  software  engineering  environments  integrate  with  the  systems 
engineering  environments  across  all  software  team  members  for  the  software  under  review 

7.  The  software  development  and  test  environments  are  established  and  have  adequate  capability 
and  capacity  to  meet  the  software  development  and  test  requirements  and  schedules 

8.  The  contractor  has  demonstrated  that  the  software  processes,  standards,  procedures,  and 
conventions  are  being  followed,  as  appropriate  to  this  point  in  the  life  cycle 

J.  Metrics  and  Technical  Performance  Measures 

1.  The  software  metrics  are  sufficient  for  meeting  the  information  needs  for  program  and 
engineering  management  and  incorporate  lessons  learned  from  the  metrics  experience  to  date 

2.  Software  metrics  are  being  collected,  analyzed,  reported,  and  used  for  management  and  technical 
decision-making,  including  risk  management,  as  appropriate  to  this  point  in  the  life  cycle 

3.  Adequate  corrective  actions  have  been  defined  to  address  the  underlying  problems  indicated  by 
software  metrics  that  are  outside  of  documented  thresholds 

4.  TPMs  are  being  collected,  analyzed,  reported,  and  used  for  managing  the  utilization  of  all  critical 
computer  resources,  e.g.,  processors,  memory,  storage,  and  input  and  output  channels  and 
networks 

5.  TPMs  are  being  collected,  analyzed,  reported,  and  used  for  managing  the  software-related  KPPs 
and  driving  requirements,  including  response  time  and  timeline  requirements 

6.  Adequate  corrective  actions  have  been  defined  to  address  the  underlying  problems  indicated  by 
software  TPMs  that  are  outside  of  documented  thresholds 

7.  The  contractor  has  demonstrated  that,  for  metrics  or  TPMs  outside  of  thresholds,  corrective 
actions  have  been  initiated,  managed,  and  tracked  to  closure 

8.  The  software  problem  and  deficiency  report  status  indicates  that  adequate  progress  is  being  made 
in  implementing  and  verifying  solutions  to  documented  problems,  and  that  the  documented 
problems  are  being  addressed  in  accordance  with  their  severity 
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30.5  Post  Review  Action 

After  completing  the  SAR,  the  contractor  shall  publish  and  distribute  copies  of  Review  Minutes.  The 
contracting  agency  officially  acknowledges  completion  of  the  SAR. 
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Appendix  D  -  Preliminary  Design  Review  (PDR) 


40.  Preliminary  Design  Review  (PDR) 

The  PDR  is  a  multidisciplinary  technical  product  and  SE  process  conducted  to  assess  whether  the 
preliminary  system  and  configuration  item  (Cl)  or  a  functionally  related  group  of  end  items  design  under 
review  are  sufficiently  mature  to  proceed  into  detailed  and  critical  design,  e.g.: 

a.  Is  consistent  with  the  Capability  Development  Document  (CDD)  as  defined 

b.  Can  meet  the  stated  performance  requirements  within  program  cost,  budget,  schedule,  risk, 
user,  and  other  constraints 

c.  Is  fully  captured  in  the  performance  specifications  for  each  configuration  item  in  the  system 
and  allocated  baseline 

d.  Has  each  function  in  the  functional  baseline  allocated  to  one  or  more  system  Cl,  consisting  of 
hardware  and  software  elements  and  deliverables 

For  complex  systems,  a  series  of  PDRs  for  each  subsystem  or  configuration  item  shall  be  conducted, 
leading  to  an  overall  system  PDR.  For  software  items  a  series  of  SARs  will  be  conducted  according  to  the 
software  development  plan  (SDP)  based  on  the  selected  life  cycle  model(s)  (see  Appendix  C).  When 
individual  reviews  have  been  conducted,  the  emphasis  of  the  overall  system  PDR  shall  focus  on 
configuration  item  functional  and  physical  interface  design,  as  well  as  overall  system  design 
requirements.  The  PDR  determines  whether  the  hardware,  human,  and  software  preliminary  designs,  to 
the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle  model(s),  are  complete,  and  whether  the 
Integrated  Product  Team  is  prepared  to  start  detailed  design  and  test  procedure  development. 

Each  PDR  shall  be  tailored  for  the  review  of  the  technical  scope  and  risk  of  the  system,  segment, 
subsystem,  and  Cl,  and  made  an  integral  part  of  the  Systems  Engineering  Plan. 

The  PDR  shall  be  conducted  only  when  all  major  hardware  design  issues  have  been  resolved  and  work 
can  begin  on  hardware  detailed  design.  For  software,  design  issues  will  be  resolved  to  the  extent  specified 
in  the  SDP  based  on  the  selected  life  cycle  model(s)  and  by  the  SAR  and  software  build  reviews. 

40.1  General 

PDRs  shall  be  held  to  assess  that  the  set  of  subsystem  or  Cl  requirements  baselined  at  SRR  and  SFR 
correctly  and  completely  implement  all  system  requirements  allocated  to  the  segment,  subsystem,  and  Cl. 
The  PDR  also  determines  whether  segment,  subsystem,  and  Cl  requirements  trace  with  the  system  design. 

A  PDR  is  held  for  each  Cl  or  aggregation  of  CIs  in  the  specification  tree.  Individual  Cl  PDRs  should 
ensure  that: 

a.  The  Cl  architecture  is  complete 

b.  The  Cl  development  specification  is  complete,  or  development  specification  is  approved 

c.  The  allocated  baseline  is  complete,  or  allocated  baseline  approved 

d.  The  software  requirements  and  architecture  are  complete  to  the  extent  specified  in  the  SDP 
based  on  the  selected  life  cycle  model(s) 

A  system  PDR  shall  be  held  after  completion  of  all  hardware  CIs  and  aggregate  of  hardware  Cl  PDRs. 
After  completion  of  all  software  SARs  specified  in  the  SDP,  and  after  the  individual  PDRs  and  software 
SARs  have  been  conducted,  the  emphasis  of  the  overall  system  PDR  shall  focus  on: 

a.  The  configuration  item  functional  and  physical  interface  design,  as  well  as  overall  system 
design  requirements 

b.  The  maturity  of  the  hardware,  human,  and  software  preliminary  designs 

c.  The  maturity  of  the  system  and  El  design,  to  demonstrate  that  the  design  has  progressed  to  the 
point  that  the  contractor  is  prepared  to  start  detailed  design  and  test  procedure  development 
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d.  The  progress,  technical  adequacy,  and  risk  resolution  (on  a  technical,  cost,  and  schedule  basis) 
of  the  selected  design  approach 

e.  The  compatibility  of  the  Hardware  Configuration  Item  (HWCI)  development  specification  with 
performance  and  engineering  specialty  requirements 

f.  The  degree  to  which  each  El  definition  has  been  baselined  and  assess  the  technical  risk 
associated  with  the  selected  manufacturing  methods  and  processes  for  each  El 

g.  The  existence  and  compatibility  of  the  physical  and  functional  interfaces  among  the 
configuration  item  and  other  items  of  equipment,  facilities,  computer  software,  and  personnel 

h.  The  evaluation  of  the  progress,  consistency,  and  technical  adequacy  of  the  Software 
Configuration  Items  (SWCIs),  to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle 
model(s),  e.g.: 

(1)  The  selected  top-level  software  design  and  test  approach 

(2)  Compatibility  between  software  requirements  and  preliminary  design 

(3)  The  preliminary  version  of  the  operation  and  support  documents 

i.  The  results  of  peer  reviews  of  requirements  and  preliminary  design  documentation 

j.  The  determination  that  the  subsystem  requirements,  subsystem  preliminary  design,  results  of 
peer  reviews,  and  plans  for  development  and  testing  form  a  satisfactory  basis  for  proceeding 
into  detailed  design  and  test  procedure  development 

The  PDR  shall  also  confirm  that  the: 

a.  Detailed  system  design  approach  (as  an  integrated  composite  of  people,  product,  and  process 
solutions)  satisfies  the  allocated  functional  baseline 

b.  Remaining  risks  are  mitigated  with  closure  plans  and  establishment  of  a  baseline  design  for  the 
system,  subcomponents,  or  support  elements 

c.  Technology  is  mature  enough  (minimum  TRL  5)  for  development  to  proceed  or  whether 
alternate  technologies  should  be  considered  primary 

40.2  Purpose 

The  purpose  of  the  PDR  is  to  evaluate  the  set  of  subsystem  requirements  and  determine  whether  they 
correctly  and  completely  implement  all  system  requirements  allocated  to  the  subsystem.  In  addition,  it 
must  be  determined  whether  subsystem  requirements  trace  with  the  system  design,  and  that  the  plans  for 
development  and  testing  form  a  satisfactory  basis  for  proceeding  into  detailed  design  and  test  procedure 
development. 

Specifically,  the  purpose  of  the  PDR  is  to  address  and  resolve  critical,  systemwide  issues,  e.g.: 

a.  Establish  a  baseline  design  for  every  Hardware  Configuration  Item  (HWCI)  and  for  every 
SWCI  as  specified  in  the  SDP  based  on  the  selected  life  cycle  model(s) 

b.  Evaluate  the  progress,  technical  adequacy,  and  risk  resolution  (on  a  technical,  cost,  and 
schedule  basis)  of  the  selected  design  approach 

c.  Determine  design  approach  compatibility  with  performance  and  engineering  specialty 
requirements  of  each  configuration  item  development  specification 

d.  Evaluate  the  degree  of  definition  and  assess  the  technical  risk  associated  with  the  selected 
manufacturing  methods  and  processes 

e.  Establish  the  existence  and  compatibility  of  the  physical  and  functional  interfaces  among  the 
configuration  items  and  other  items  of  equipment,  facilities,  computer  software,  and  personnel 

f.  For  SWCIs,  this  review  will  focus  on: 
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(1)  The  evaluation  of  the  progress,  consistency,  and  technical  adequacy  of  the  selected  top- 
level  design  and  test  approach 

(2)  The  compatibility  between  software  requirements  and  preliminary  design,  and  on  the 
preliminary  version  of  the  operation  and  support  documents 

40.3  Objective 

The  objective  of  the  PDR  is  for  the  Contractor  and  the  Acquisition  agency  to: 

a.  Review  any  changes  to  the  functional  baseline  (FBL) 

b.  Review  and  confirm  the  functional  architecture 

c.  Review  and  confirm  the  physical  hierarchy 

d.  Review  and  confirm  the  allocated  baseline  for  each  configuration  item,  including  the 
completeness  and  compatibility  of  interfaces  between  the  items  and  between  the  items  and 
other  systems,  facilities,  and  personnel 

e.  Review  and  confirm  the  two-way  traceability  from  the  source  of  the  FBL  to  the  allocated 
baseline  and  back 

f.  Review  and  confirm  and  verify  that  the  allocated  baseline  can  meet  the  system  requirements 

g.  Review  and  confirm  the  validity  of  the  updated  risk  assessment  for  system  development  and 
demonstration 

h.  Review  and  validate  the  basis  and  the  balance  between  performance,  cost,  schedule,  and  risk  for 
each  element  in  the  architectures  and  each  requirement  in  the  baseline 

i.  Review  and  validate  that  the  contractor’s  system- allocated  baseline  supports  the  updated  cost 
analysis  requirements  description  (CARD) 

j.  Review  and  validate  the  updated  program  schedule,  including  system  and  software  critical  path 
drivers 

k.  Review  and  validate  the  approved  SWCI 

Additionally,  an  assessment  will  be  conducted  on  each  prototype  (as  applicable  and  specific  to  each 
development  program)  to: 

a.  Evaluate  the  progress,  technical  adequacy,  and  risk  resolution  of  the  selected  design  approach 

b.  Determine  its  alignment  with  the  evolving  FBL  and  architecture 

c.  Demonstrate  the  compatibility  of  the  allocated  baseline  with  the  physical  and  functional 
interfaces  and  other  items,  facilities,  and  personnel 

40.4  PDR  “Acceptance  Criteria” 

At  PDR  all  major  systems  engineering  management  elements  and  activities  that  are  program  risk  drivers 
are  considered.  “Completion”  when  used  in  reference  to  software  indicates  that  the  activity  or  product  is 
complete  to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle  model(s). 

The  intent  of  the  PDR  is  to  ascertain  that: 

a.  The  hardware  functional  decomposition  has  been  completed 

b.  An  accurate,  comprehensive  allocation  baseline  has  been  approved 

c.  The  hardware  baseline  design  has  been  established 

d.  The  software  architectural  design  has  been  completed  to  the  extent  specified  in  the  SDP  based 
on  the  selected  life  cycle  model(s) 

In  preparation  for  and  scheduling  of  a  PDR,  the  contractor  shall  demonstrate  to  the  satisfaction  of 
the  contracting  agency  that: 
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a.  The  contractor  has  appropriately  addressed  the  requirements  criteria  elements 

b.  All  applicable  engineering  activities  have  been  properly  conducted  in  support  of  the  criterion 

c.  A  viable  technical  and  program  risk  management  strategy  have  been  demonstrated  to  the 
satisfaction  of  the  contracting  agency 

d.  Effective  and  efficient  technical  progress  is  made  towards  meeting  all  cost,  schedule,  and 
technical  performance  requirements 

The  PDR  “Acceptance  Criteria”  shall  be  organized  under  the  following  five  major  categories: 

1 .  Systems  Engineering  and  Architecture  Development  (40.4. 1) 

2.  System,  Segment,  and  Subsystem  Design  (40.4.2) 

3.  System  Verification  and  Validation  (40.4.3) 

4.  Engineering  Disciplines  and  Specialty  Engineering  (40.4.4) 

5.  Integrated  Technical  Risk  and  Mitigation  (40.4.5) 

This  review  shall  serve  as  objective  evidence  of  the  contractor’s  technical  effort  that  supports  the  basic 
and  agreed-to  PDR  “Acceptance  Criteria,”  e.g.: 

a)  Does  the  status  of  the  technical  effort  and  design  indicate  operational  test  success 
(operationally  suitable  and  effective)? 

b)  Can  the  preliminary  design,  as  disclosed,  satisfy  the  Capability  Development  Document? 

c)  Has  the  system-allocated  baseline  been  established  and  documented  to  enable  detailed  design 
to  proceed  with  proper  configuration  management? 

d)  Are  adequate  processes  and  metrics  in  place  for  the  program  to  succeed? 

e)  Have  human  integration  design  factors  been  reviewed  and  included,  where  needed,  in  the 
overall  system  design? 

f)  Are  the  risks  known  and  manageable  for  development  testing  and  operational  testing? 

g)  Is  the  program  schedule  executable  (technical  and  cost  risks)? 

h)  Is  the  program  properly  staffed? 

i)  Is  the  program  executable  with  the  existing  budget  and  with  the  approved  system-allocated 
baseline? 

j)  Does  the  updated  cost  estimate  fit  within  the  existing  budget? 

k)  Is  the  preliminary  design  producible  within  the  production  budget? 

l)  Is  the  updated  cost  analysis  requirements  description  consistent  with  the  approved  allocated 
baseline? 

m)  Is  the  software  functionality  in  the  approved  allocated  baseline  consistent  with  the  updated 
software  metrics  and  resource-loaded  schedule? 

n)  Are  the  verification  plans  and  resources  in  place  to  continue  to  CDR? 

The  primary  PDR  data  is  the  Decision  Data  Base  (DDB)  documenting  or  demonstrating  these  items. 

40.4.1  Systems  Engineering  and  Architecture  Development 

Evidence  of  Systems  Engineering  and  Architecture  Development  requirements  maturity  criteria  examples 
at  PDR: 

A.  Systems  Engineering  Review  Criteria 

1.  The  system,  segment,  subsystem,  and  component  allocated  requirements  are  complete,  feasible, 
verifiable,  and  clearly  stated,  e.g.: 
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a.  Preliminary  System  Design  is  correlated  with  and  reflected  in  the  Allocated  Requirements 
Baseline 

b.  Preliminary  design  of  the  system,  segments,  subsystems,  and  components  is  correlated  with 
the  system  architecture  views  and  descriptions  and  is  traceable  to  the  Functional  and 
Allocated  Baselines 

c.  Preliminary  design  considers  end-to-end  processing  capabilities  for  the  system,  segments, 
subsystems,  and  components  architectures,  including  timelines  and  capacities  for  production, 
integration,  operations,  maintenance,  training,  demilitarization,  and  disposal 

d.  Preliminary  design  data  (e.g.,  drawings,  specifications,  etc.)  for  the  system,  segments, 
subsystems,  and  components  are  complete  and  placed  under  configuration  control 

e.  End-to-end  data  flow  for  the  system  is  complete 

f.  Preliminary  design  HW  and  SW  prototypes,  and  their  analyses  and  results,  are  documented 
and  placed  under  configuration  control 

g.  All  external  dependencies  are  identified  and  documented 

2.  System  Requirements  Functional  Decomposition  Completed,  e.g.: 

a.  Requirements  flowdown  and  derivation  from  system  to  segment  and  from  segment  to 
subsystem  are  complete  and  traceable  (no  TBDs,  TBSs,  and  TBRs) 

b.  Requirements  flowdown  and  derivation  from  subsystem  to  component  are  complete  and 
traceable  (all  TBDs,  TBSs,  TBRs,  and  deferrals  are  identified) 

c.  Requirements  flowdown  and  derivation  for  intersegment  and  inter-subsystem  interfaces  are 
complete  and  traceable  (all  TBDs,  TBSs,  TBRs,  and  deferrals  are  identified) 

d.  Design-to  allocated  requirements  for  the  system,  segments,  and  subsystems  are  validated  by 
specialty  engineering 

e.  Functional  flow  block  diagrams  (FFBDs)  are  completed  for  the  system,  segments, 
subsystems,  hardware  components,  and  intersegment,  and  inter-subsystem  interfaces, 
demonstrating  flowdown  and  traceability  between  higher-  and  lower-level  allocated 
requirements 

f.  The  system  and  segment,  subsystem,  and  component  design  specifications  are  under 
configuration  management  without  any  major  TBDs  or  open  items 

g.  Preliminary  long-lead  production  requirements  are  developed  and  documented 

3.  Allocated  baseline  established  is  based  on  and  traceable  to  the  approved  Mission  and  System 

Functional  Baselines,  e.g.: 

a.  Allocated  baseline  is  consistent  with  the  physical  hierarchy  and  design-to  functional 
performance  requirements  for  all  products  in  the  hardware  hierarchy 

b.  System  functional  and  performance  requirements  are  allocated  to  all  system  segments, 
subsystems,  and  components,  e.g.,  system,  segment,  subsystem,  and  component-level 
allocation  performance  analyses  are  completed  and  traceable  to  accepted  trade  results 

c.  Interoperability  functional  performance  requirements  are  allocated  to  all  system,  segment, 
and  subsystem  preliminary  designs 

B.  Interoperability  Architecture 

1 .  Demonstrate  that  the  system  design  satisfies  the  operational  architecture 

2.  The  system  design  identifies  all  operational  nodes  (OV-2)  and  associated  connectivity,  e.g.,  is 

able  to  address  systems  that  constitute  the  operational  nodes  such  as  satellites,  ground  antennas, 

command  and  control  equipment,  mission  data  user  equipment,  etc. 
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3.  The  system  design  demonstrates  it  provides  required  information  to  the  organizational 
relationships  between  system  operators  and  users  described  by  OV-4 

4.  The  system  design  identifies  systems  and  subsystems  that  support  all  operational  activities  during 
the  entire  life  cycle  from  acquisition  to  EOL  operations  described  by  OV-5 

5.  The  system  design  is  able  to  provide  traceability  between  operational  activities  and  system 
functions  (SV-5) 

6.  The  system  design  reflects  a  complete  set  of  data  exchanges  necessary  between  internal  and 
external  interfaces  (SV-6) 

7.  System  design  interfaces  incorporate  the  set  of  DISR  interoperability  standards  shown  in  TV-1; 
all  unique  interfaces,  data  formats,  etc.,  have  been  approved  by  the  contracting  agent 

8.  The  system  design  is  consistent  with  NCOW-R,  KIP  Compliance,  and  IA  Compliance 

C.  All  design  trade  studies  include  LCC  and  CAIV  analyses  results  supporting  the  allocated  technical 

and  functional  baselines 

1 .  Results  of  LCC  and  CAIV  analyses  include  sensitivity  of  allocated  performance  parameters  to 
cost 

2.  LCC  and  CAIV  models  representing  planned  and  approved  program  development,  operational, 
and  sustainment  costs  are  baselined,  including  cost  impacts  to  other  “external”  systems 

3.  LCC  and  CAIV  modeling  and  analyses,  as  applied  and  correlated  with  each  SW  design,  depict 
projected  program  development,  and  operational  and  sustainment  costs,  as  well  as  projected  cost 
impacts  to  other  “external”  systems 

4.  LCC  and  CAIV  methodology  is  presented,  and  demonstrates  that  valid  trade  studies  were 
conducted 

D.  System  integration  and  verification  functional  performance  requirements  are  allocated  to  all 

segments,  subsystems,  and  components 

1.  Segment,  subsystem,  and  component-level  verification  planning  is  completed  with  rationale  for 
verification  objectives,  types,  levels  and  sequence  of  verification,  venues,  and  verification  data  to 
be  collected 

2.  Segment,  subsystem,  and  component-level  integration  and  test  planning  are  completed  with 
rationale  for  test  objectives,  type,  levels  and  sequence  of  testing,  test  venues,  and  test  data  to  be 
derived 

3.  Processes  and  procedures  are  developed  for  system  integration  and  verification 

4.  Preliminary  processes  and  procedures  are  defined  for  segment,  subsystem,  and  component 
integration  and  verification 

5.  Segment,  subsystem,  and  component-level  cross-reference  requirements  are  baselined  and 
completed 

E.  System,  segment,  subsystem,  and  component-level  interfaces  are  baselined 

1.  Preliminary  internal  (segment-to-segment,  subsystem-to-subsystem  and  component-to- 
component)  interfaces  are  designed  and  placed  under  configuration  control 

2.  External  interfaces  (system,  system  of  systems,  and  family  of  systems)  design  is  completed  and 
put  under  configuration  control 

3.  All  physical  and  functional  interfaces  between  HWCI  and  other  items  of  equipment,  software  and 
firmware,  and  facilities  are  defined  and  documented 

4.  All  interfaces  between  SWCI  and  other  configuration  items  (both  internal  and  external)  are 
defined  and  documented 
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F.  Allocated  decomposition  is  completed  for  each  HWCI  and  SWCI 

1.  Decomposition  for  HWCIs  and  SWCIs  is  traceable  to  the  requirements  and  functional  baselines 

2.  Allocated  decomposition  for  all  HWCIs  and  SWCIs  is  under  configuration  control,  e.g.,  all 
changes  to  the  requirements  and  functional  baselines  are  identified,  tracked,  and  documented 

3.  The  preliminary  physical  (also  known  as  (a.k.a.)  product)  baseline  is  developed  for  all  HWCIs 
and  SWCIs 

G.  System  performance  (design)  specification  is  traceable  to  the  allocation  baseline,  e.g.,  segment, 
subsystem,  and  component  specifications  are  developed  and  traceable  to  the  system  performance 
specification 

40.4,2  System,  Segment,  and  Subsystem  Design 

Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  maturity  criteria  examples  at  PDR: 

A.  System,  segment,  subsystem,  and  component  preliminary  design  is  completed  and  baselined. 

1.  The  preliminary  design  demonstrates  traceability  among  all  considerations,  i.e.: 

a.  Between  allocated  requirements,  engineering  trade  study  results,  technology  selections,  and 
technical,  programmatic,  schedule,  and  cost  risks 

b.  The  adequacy  of  the  preliminary  design  has  been  demonstrated  using  ongoing  engineering 
analyses,  considering  all  relevant  specialty  engineering  disciplines,  e.g. : 

(1)  Engineering  analyses  adequately  support  the  allocated  decomposition  of  requirements  to 

hardware  and  software  items  for  system  segments,  subsystems,  and  components 

(2)  Engineering  analyses  results  adequately  demonstrated  the  readiness  of  the  design  to 
proceed  to  CDR 

2.  Demonstrate  that  the  preliminary  design  is  traceable  to  and  correlated  with  all  critical  allocated 
requirements 

3.  Appropriate  margins  and  allowances  are  established  at  the  segment,  subsystem,  and  component 
levels 

4.  Design  development  planning  is  completed  and  baselined,  e.g.,  preliminary  design  drawings  are 
under  configuration  control 

5.  Preliminary  electrical,  mechanical,  and  functional  performance  schematics  are  available, 
including  Functional  Flow  Block  Diagrams  (FFBDs)  for  inter-  and  intra-segments  and 
subsystems 

6.  Preliminary  GSE-identified  and  preliminary  design  concepts  are  developed  traceable  to  the 
system  functional  and  allocated  requirements  baselines  and  to  the  system  architecture 

7.  Critical  components  of  the  system  are  identified 

B.  C4I  allocations  are  incorporated  into  the  preliminary  design  across  segments,  subsystems,  and 
components 

1.  Allocations  include  battle  management  and  information  technology  (IT)  needs,  dependencies,  and 
interrelationships  between  system  segments,  subsystems,  and  the  system,  system  of  systems,  and 
family  of  systems 

2.  Allocations  ensure  C4I  interoperability,  interconnectivity,  supportability,  synchronization,  and 
sufficiency 

C.  Preliminary  design  is  correlated  with  the  threat  scenarios  and  threat  environment  parameters,  e.g.: 
threat  scenario  operational  and  environmental  allocations  are  incorporated  into  the  preliminary  design 
traceable  to  all  segments,  subsystems,  and  components 
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D.  Environmental,  e.g.,  natural,  thermal,  humidity,  transport  parameters  are  correlated  with  the 
preliminary  design 

E.  Environmental  allocations  incorporated  into  the  preliminary  design  are  traceable  to  all  segments, 
subsystems,  and  components 

F.  Reliability,  availability,  maintainability,  and  testability  (RAM&T)  allocated  requirements  are 
incorporated  into  the  preliminary  design,  e.g.,  RAM&T  allocations  are  traceable  to  segments, 
subsystems,  and  components 

G.  System  operational  sustainment  key  performance  parameters  are  incorporated  into  the  preliminary 
design,  including  all  major  system  and  program  requirements,  e.g.,  the  LCC  sustainment  model  is 
correlated  with  the  preliminary  design 

H.  Risk  mitigation  solutions  in  the  system  risk  model  are  traceable  to  and  correlated  with  the  preliminary 
design 

I.  Ongoing  Industrial  Base  (IB)  assessment  results  are  correlated  with  the  preliminary  design;  new  risk 
areas  (not  identified  at  SFR)  are  prioritized  and  the  mitigation  processes  defined,  including  resources 
and  schedule  requirements,  e.g.: 

1.  IB  assessment  data  (e.g.,  DMSMS  [Diminishing  Manufacturing  Sources  and  Material  Shortages], 
parts  obsolescence)  is  correlated  with  identified  and  implicit  design  risk  areas 

2.  Mitigation  strategies  are  planned  and  implemented,  including  resources  and  schedule 
requirements 

J.  Key  allocated  performance  requirements  are  traceable  to  the  preliminary  system  design  for  all  major 
subsystems  and  components,  e.g.: 

1.  All  major  subsystem  and  component  allocations  are  incoiporated  into  the  preliminary  design 

2.  Key  parameters  and  information  (developed  and  assessed  at  SFR)  are  implemented  for  each 
major  subsystem  and  component  preliminary  design,  e.g.: 

a.  Major  performance  parameters  are  incoiporated 

b.  Critical  technologies  are  under  development 

c.  Critical  design  and  manufacturing  requirements  and  challenges  (identified  at  SFR)  are 
correlated  with  preliminary  design 

Note:  The  following  examples  are  intended  to  provide  clarification  of  the  types  of  data  and  level  of  detail 
expected  to  be  addressed  at  PDR.  It  is  intended  that  the  contractor  will  identify  those  subsystems  and 
components  applicable  to  the  type  of  system  being  developed  and  the  appropriate  criteria  for  each 
subsystem  and  component  necessary  to  effectively  evaluate  and  assess  the  preliminary  system  design  and 
its  technical,  cost,  and  schedule  parameters,  and  demonstrate  that  the  design  includes  identification  and 
recovery  from  known  failure  modes,  i.e.: 

=>  For  Electrical  Power: 

•  Preliminary  Electrical  Power  Distribution  System  (EPDS)  performance  requirements, 
characteristics,  and  operational  criteria  are  defined,  including  initial  power  budgets,  total 
power  demand  with  allowable  margins,  and  modes  of  operation  (frequency  and  duration) 

•  Preliminary  selection  and  evaluation  of  the  type(s)  of  power  supply  sources  being  considered, 
their  specific  technology  and  topology 

•  Preliminary  battery  (or  energy  storage)  power  requirements  are  identified  and  modes  of 
operation  defined  (frequency  and  duration) 

•  Battery  life  requirements  (BOL  and  EOL)  and  other  unique  requirements  that  may  impact 
battery  selection  or  design 

•  Candidate  battery  cell  technologies  are  identified  and  battery  architectures  defined 
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=^>  For  Software: 

•  The  “Acceptance  Criteria”  for  software  detailed  in  Appendix  C,  sections  30.4  SAR 
“Acceptance  Criteria”  (paragraphs  A,  B,  C,  D,  F,  G,  H,  I,  and  J)  are  satisfied  to  the  extent 
specified  in  the  SDP  based  on  the  selected  life  cycle  model(s). 

40.4.3  System,  Segment,  and  Subsystem  Verification  and  Validation 

Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  V&V  requirements  maturity  criteria 
examples  at  PDR: 

A.  System,  Segment,  Subsystem,  and  hardware  Component  V&V  approaches  are  developed  for  the 
preliminary  design 

1.  Preliminary  design  demonstrates  that  major  system,  segment,  subsystem,  and  hardware 
component-allocated  requirements  can  be  verified  and  validated,  e.g.: 

a.  V&V  approaches  are  developed  for  the  preliminary  design  address  system  of  systems, 
system,  segment,  subsystem,  and  hardware  component  levels 

b.  V&V  approaches  include  analytical,  modeling  and  simulation,  and  testing  processes  and 
procedures  for  preliminary  design 

c.  V&V  processes  and  procedures  address  new  technology,  verification,  and  qualification 
technical  practices,  system-level  demonstrations  and  tests,  support  required  from  external 
organizations  and/or  facilities,  and  resource  requirements  for  the  preliminary  design 

d.  V&V  processes  and  procedures  for  the  preliminary  design  are  based  on  proven,  referenced 
practices 

e.  Updated  subsystem  and  component  VCRMs  are  complete  and  consistent  with  system  and 
segment  allocated  requirements  and  internal  and  external  interface  allocated  requirements, 
e.g.: 

(1)  Segment,  subsystem,  and  component  VCRMs  are  traceable  to  the  system  VCRM 

(2)  The  updated  subsystem  and  component  VCRMs  are  baselined  and  under  configuration 
management 

(3)  V&V  methods  in  the  system,  segment,  subsystem,  and  hardware  component  VCRMs  are 
adequate  to  verify  the  system  and  its  segment,  subsystem,  and  hardware  component 

f.  The  completion  of  software  test  and  qualification  plans  and  the  allocation  of  requirements  to 
tests  are  detailed  in  Appendix  C,  30.4  SAR  “Acceptance  Criteria”  paragraph  E 

B.  System  operational  functions  and  environments  for  the  preliminary  design  are  traceable  to  the 
contractor’s  operations  concept  (OpsCon)  and  the  allocated  baseline 

1.  Demonstrate  that  the  system  V&V  test  environment  allocations  are  traceable  to  the  system 
performance  specification  for  the  preliminary  design 

2.  Demonstrate  that  the  preliminary  design  is  correlated  with  and  traceable  to  all  initially  identified 
allocated  and  physical  environmental  parameters,  verification  approaches,  and  processes 

C.  DT&E  elements  are  correlated  to  the  preliminary  design 

D.  OT&E  allocated  requirements  are  incorporated  into  the  preliminary  design 

E.  Test  bed(s)  and  test  facilities  chosen  based  on  the  preliminary  design  are  deemed  adequate  to  perform 
system,  segment,  subsystem,  and  interface  requirements  verification,  e.g.:  for  critical  hardware  and 
software  items,  arrangements  for  procuring  and/or  scheduling  the  use  of  V&V  resources  (simulators, 
test  beds,  test  facilities)  have  been  demonstrated 

F.  Test  requirements  and  test  data  collected  to  date  for  the  preliminary  design  are  traceable  to 
operational  requirements  via  specifications  and  V&V  cross-reference  matrices  (VCRMs),  e.g.,  use  of 
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comparative  test  data  to  anchor  representative  system,  segment,  subsystem  models,  and  simulations  to 
real-world  environments  and  allocated  requirements  are  demonstrated 

G.  V&V  risk  approaches,  processes,  and  procedures  are  developed  for  the  preliminary  design 

H.  V&V  test  deficiencies,  including  those  based  on  technology  deficiencies  established  at  SFR,  are 
correlated  with  the  preliminary  design  and  the  impact  assessed 

I.  Risk  mitigation  approaches  developed  and  integrated  into  the  system  risk  model  at  SFR,  including 
V&V  resource  requirements  are  correlated  with  the  preliminary  design.  Software  risk  management 
activities  are  detailed  in  paragraph  30.4  SAR  “Acceptance  Criteria”  paragraph  G  of  Appendix  C 

40.4.4  Engineering  Disciplines  and  Specialty  Engineering 

Evidence  of  Engineering  Discipline  and  Specialty  Engineering  identification  and  assessment  maturity 
criteria  (categories  listed  in  A  through  R  below)  at  PDR  in  terms  of,  e.g.: 

1 .  Key  performance  requirements 

2.  Key  performance  parameters 

3.  Use  of  heritage  systems,  components,  and  technology 

4.  Use  of  new  designs 

A.  Parts,  Materials,  and  Processes  (PM&P) 

1.  PM&P  allocated  requirements  are  incoiporated  into  the  preliminary  design 

2.  Environments  and  environmental  parameters  impacting  parts  performance  are  incorporated  into 
the  preliminary  design 

3.  Parts  engineering  design  analyses  are  completed  for  the  preliminary  design  addressing  risk 
assessments,  long-lead  items,  technologies,  sources  of  supply,  and  the  common  quality  levels 
(i.e.,  reliability)  of  the  parts 

4.  Results  of  preliminary  design  analyses  are  used  to  develop  preliminary  parts  list 

B.  Test  and  Evaluation  (T&E) 

1.  Initial  T&E  planning  is  traceable  to  the  preliminary  design  correlating  all  test  objectives,  test 
environments,  and  test  resources  with  allocated  requirements 

2.  Selected  T&E  approaches  are  correlated  with  the  preliminary  design,  e.g.: 

a.  Test  approaches  are  developed  into  preliminary  test  processes  and  procedures  for  verifying 
the  system,  segments,  subsystems,  and  components-allocated  requirements 

b.  T&E  processes  and  procedures  capture  the  characteristics,  effectivity(s),  and  margins  for  each 
particular  test  item 

3.  Test  and  verification  data  gathering,  reduction,  and  analysis  processes  for  the  preliminary  design 
are  developed,  including  test  environment(s),  operations,  procedures  to  be  performed,  data 
acquisition  requirements,  documentation,  methods  of  analysis,  and  pass-fail  (i.e.,  success)  criteria 

C.  Survivability  and  Vulnerability 

1.  Survivability  and  vulnerability  threat  allocations  incoiporated  into  the  preliminary  design  are 
traceable  to  the  categories  of  expected  threats,  threat  environments,  and  their  likelihood  of 
occurrence 

2.  System  and  threat  interaction  analyses  are  completed;  threat  margins  are  established  and 
baselined  for  the  preliminary  design 

3.  Survivability  design  solutions  are  correlated  with  and  incoiporated  into  the  preliminary  design  to 
mitigate  each  known  threat 

D.  Environmental  Safety  and  Occupational  Health  (ES&OH) 
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1.  Life  cycle  environmental  allocations  are  incorporated  into  the  preliminary  design 

2.  Data  compiled  for  PDR  Programmatic  ES&OH  Evaluation  (PESEIE)  compliance  objectives  is 
correlated  with  the  preliminary  design,  including  an  assessment  of  the  interrelationships  and 
interdependency  of  the  operational  environments 

3.  Elazardous  materials  management  and  pollution  prevention  processes  and  procedures  are 
developed  and  correlated  with  the  preliminary  design 

4.  Critical  human  safety  and  health  factors  are  correlated  with  the  preliminary  design 

E.  Mass  Properties 

1.  Mass  properties  margins  (average  or  complex)  are  established  for  PDR  and  correlated  with  the 
preliminary  design,  including  allowable  growth  allocations  and  metrics 

2.  Calculated  weight  growth,  center  of  gravity,  and  moments  of  inertia  parameters  are  allocated  to 
the  preliminary  design 

F.  System  Security  Engineering  (SSE)  Communications  Security  (COMSEC),  Information  Assurance 
(IA),  and  Program  Protection  (PP): 

1.  SSE,  COMSEC,  IA,  and  PP  security  requirements  are  allocated  and  incorporated  into  the 
preliminary  design  IAW  DoD  and  AF  policies,  directives,  and  system  specifications 

2.  Implementation  of  program  protection  countermeasures  is  addressed 

3.  SSE,  COMSEC,  IA,  and  PP  requirements  based  on  updated  threat,  vulnerability,  risk,  and 
countermeasure  assessments  are  addressed  in  preliminary  design 

4.  Information  Assurance  requirements  are  included  in  the  preliminary  system  design  along  with 
certification  and  accreditation  requirements  and  schedules  using  the  DIACAP 

5.  Program  baseline  costs  for  SSE,  COMSEC,  IA,  and  PP  implementation  and  sustainment  are 
updated 

G.  Interoperability 

1 .  Allocated  system  and  mission  interoperability  requirements  are  incoiporated  into  the  preliminary 
design 

2.  Allocated  requirements  from  new  and  unique  standards  approved  for  inclusion  in  DISR  (i.e.,  new 
data  formats,  interdependency,  data  exchange  protocols  and  schemas,  Ethernet  alternatives)  are 
correlated  with  and  incoiporated  into  the  preliminary  design 

3.  Allocated  interoperability  requirements  for  all  interrelationships  and  interdependency  are 
incorporated  into  the  preliminary  design 

H.  Reliability  and  Maintainability  (R&M) 

1.  R&M  allocated  requirements  are  incorporated  into  the  preliminary  design 

2.  R&M  analyses  results  are  correlated  with  the  preliminary  design,  e.g.: 

a.  Approaches  and  processes  are  developed  for  implementing  environmental  and  thermal  stress 
screening  (ESS  and  TSS)  for  the  preliminary  design 

b.  Packaging,  Handling,  Storage,  and  Transportability  (PHS&T)  environmental  allocated 
requirements  in  the  R&M  program  are  incoiporated  into  the  preliminary  design 

3.  Conduct  hardware  FMECA  for  the  preliminary  system  design  at  the  appropriate  level  (e.g.,  box- 
pin  level  or  piece-part  level),  e.g.: 

a.  Justify  that  hardware  FMECA  level  is  consistent  with  the  intended  usage  for  the  results 

b.  Demonstrate  that  hardware  FMECA  is  consistent  with  the  equipment  physical  construction 
and  the  analysis  is  supported  by  circuit  schematics 
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c.  Identify  methods  for  detecting  the  postulated  failure  modes  for  ground  test  and  for  operation 
and  identify  possible  means  for  failure  mitigation 

d.  Prepare  critical  items  list  and  single -point  failures  list 

e.  Identify  any  safety  issues  and  associated  analyses  as  appropriate,  including  system  reliability 
for  EOL  disposal 

I.  EMI  and  EMC 

1.  Electromagnetic  interference  control  processes  and  procedures  are  developed  for  the  preliminary 
design 

2.  Internal  and  external  EMI  and  EMC  allocated  requirements  are  incorporated  into  the  preliminary 
design 

3.  EMI  susceptibility  allocated  requirements  and  constraints  are  incorporated  into  the  preliminary 
design 

4.  EMI  and  EMC  critical  environmental  characteristics  and  sensitive  elements  are  correlated  with 
the  preliminary  design 

J.  Human  Systems  Integration  (HSI) 

1.  User  interface  hardware  and  software  allocated  requirements  for  operators,  users,  maintainers, 
and  sustainers  are  incorporated  into  the  preliminary  design 

2.  Usability,  maintainability,  operability,  and/or  supportability  allocated  requirements  decomposed 
from  system  functional  requirements  are  incorporated  into  the  preliminary  design 

3.  Operational  manning,  workload,  and  skill  level  allocated  requirements  are  incorporated  into  the 
preliminary  design,  e.g.,  user  interface  is  consistent  with  the  scenarios  in  CONOPS 

K.  Manufacturing  and  Producibility 

1.  Manufacturing  and  producibility  approaches  and  processes  are  developed  and  correlated  with  the 
preliminary  design 

2.  Producibility  approaches  selected  for  the  preliminary  design  demonstrate  that  the  manufacturing 
processes  that  were  chosen  support  the  preliminary  design 

L.  Life  Cycle  Logistics 

1.  Supportability  allocated  requirements  are  incorporated  into  the  preliminary  design 

2.  System-level  logistics  elements  are  correlated  with  the  preliminary  design,  including  design 
interface,  supply  support,  test  equipment,  manpower  and  personnel,  training  and  training 
equipment,  PHS&T,  facilities,  computer  resources,  technical  data,  and  maintenance  planning 

3.  Logistics  Management  Information  (LMI)  is  completed  and  validated  in  support  of  the  allocated 
baseline  for  the  preliminary  design 

M.  System  Safety 

1.  System  safety  allocated  requirements  are  incorporated  into  the  preliminary  design 

2.  Segment  and  subsystem  hazard  analyses  are  completed,  and  an  updated  balanced  list  of 
prioritized  safety  hazards  are  established  for  the  test,  operation,  and  disposal  of  the  preliminary 
design 

3.  Critical  human  safety  and  health  factors  are  correlated  with  the  preliminary  design 

4.  The  baselined  hazardous  materials  list  (compiled  and  prioritized  at  SFR)  are  correlated  with  and 
updated  as  required  for  the  preliminary  design 

N.  Contamination  Control 

1 .  Contamination  control  processes  and  procedures  are  developed  for  the  preliminary  design 
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2.  Material  outgassing  survey  results  (from  SFR)  are  correlated  with  and  updated  as  required  for  the 
preliminary  design 

O.  Quality  Assurance 

1.  Quality  and  product  assurance  allocated  requirements  are  correlated  with  and  incoiporated  into 
the  preliminary  design 

2.  Verification,  inspection,  and  test  processes  and  procedures  are  developed  for  the  preliminary 
design 

P.  Environmental  Considerations 

1 .  Environmental  study  results  (from  SFR)  are  correlated  with  the  preliminary  design 

2.  Environmental  test  and  evaluation  approaches  and  processes  are  developed  for  the  preliminary 
design 

3.  Reliability-environmental  allocation  requirements  are  incorporated  into  the  preliminary  design 

Q.  Software 

1.  Evidence  of  Software  Engineering  Discipline  and  Specialty  Engineering  identification  and 
assessment  maturity  criteria  are  detailed  in  Appendix  C,  30.4  SAR  “Acceptance  Criteria” 

R.  Data  Storage  (Security,  Access,  Distribution,  and  Delivery) 

1.  Storage  system  capability,  flexibility,  scalability,  and  preliminary  design  maturity,  e.g.: 

a.  Analysis  identifies  needed  reliability,  maintainability,  and  availability  characteristics  of 
storage  systems  environments 

b.  Capacity,  flexibility,  and  extensibility  parameters  have  been  completely  identified  that 
address  system  design  life 

c.  Key  system  components  have  been  fully  identified.  Plans  for  redundancy  are  in  place  and 
fully  identified,  including  storage  media  hardware  and  software  capabilities  and  types. 

d.  Needs  for  storage  system  management  and  performance  optimization  (including  software 
management  tools  to  provide  appropriate  partitioning  and  addressability)  are  completely 
identified 

e.  Analysis  has  fully  identified  the  operational  environments  under  which  the  storage  system 
must  operate.  Identification  of  hardening  aspects  that  must  be  addressed  is  fully  described. 

2.  Storage  System  Architecture,  e.g.: 

a.  The  Storage  System  Architecture  fully  addresses  elements,  including  communications  and 
processing  capacity 

b.  The  types  of  storage  system  needs  are  identified  and  fully  integrated  into  the  architecture. 
This  includes  items  such  as  centralized  vs.  distributed  storage;  online,  near-line,  and  offline 
needs;  archive  (including  hierarchical  storage  management,  if  appropriate),  backup,  and 
restore;  and  data  replication. 

c.  Storage  hardware  components  such  as  RAID,  Storage  Area  Networks  (SANs),  Network 
Attached  Storage  (NAS),  and  Direct  Attached  Storage  (DAS)  have  been  identified  and  fully 
integrated  into  the  architecture 

d.  Data  management  software  capabilities  have  been  identified  and  fully  integrated  into  the 
architecture.  This  includes  items  such  as  automatic  file  migration  and  transparent  file 
retrieval;  migration  between  hierarchical  levels;  and  utilities  to  report  on  media  usage,  error 
detection,  and  identification  of  media  to  be  replaced. 

3.  Security,  e.g.: 
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a.  The  level  of  user  integrity  (e.g.,  access  control  lists)  has  been  identified  that  enables  the 
system  requirements  to  be  met 

b.  The  level  of  encryption  needed  has  been  identified  that  enables  the  system  requirements  to  be 
met 

c.  The  need  for  specialized  security  capabilities,  such  as  CDS,  MLS,  and  Security  Enclaves,  has 
been  identified  and  is  included  in  the  storage  system  so  as  to  ensure  that  the  system 
requirements  are  met 

4.  Data  Distribution  Methods,  e.g.: 

a.  A  complete  list  of  data  receivers  has  been  drawn  up  to  include  both  computer  and  human 
agents 

b.  The  method(s)  of  distributing  data  to  the  various  receivers  has  been  identified.  Such  methods 
may  include  Subscribe  and  Publish,  Push  and  Pull,  and  global  or  restricted  Web-based  access. 

c.  The  data  distribution  methods  are  fully  integrated  into  the  storage  architecture  and  will  enable 
the  system-level  requirements  to  be  met 

5.  Functionality,  e.g.: 

a.  Analysis  has  fully  identified  the  physical  aspects  of  the  functionality  that  may  be  needed  to 
support  the  mission 

b.  The  types  of  platforms  (server  and  client)  and  operating  systems  supported  have  been  fully 
identified 

c.  The  data  connection  and  transport  protocols  (e.g.,  fiber  channel,  infmiband,  SWCI)  have  been 
fully  identified  and  integrated  into  the  system  architecture,  enabling  the  system-level 
requirements  to  be  met 

d.  Specific  reporting  (e.g.,  usage)  and  maintenance  metrics  (e.g.,  MTBF  and  MTTR)  have  been 
defined.  Preliminary  mapping  between  metrics  and  system-level  requirements  has  been 
completed. 

40.4.5  Integrated  Technical  Risk  Management  and  Mitigation 

Evidence  of  technical  risk  management  (RM)  process  criteria,  with  an  increased  level  of  fidelity  and 
maturity  of  identified  risk  items  and  elements,  is  provided  for  the  recommended  system  design  as  a  key 
component  of  an  integrated  program  (technical,  cost,  schedule,  and  performance)  RM  and  Mitigation 
(RM&M)  process  for  PDR. 

A.  RM&M  data  supports  PDR  level  maturity,  e.g. : 

1.  Significant  program-level  risks  impacting  technical  and  performance,  cost,  and  schedule 

2.  Risk  management  database  and  tools  for  risk  metrics  collection,  analysis,  tracking,  and  reporting 

3.  Risk  mitigation  and  reduction  strategies,  bum-down  plans  that  are  linked  to  dependencies  to  the 

Program  IMP,  IMS,  and  WBS 

4.  Continuous  risk  monitoring  and  review,  identification,  assessment,  and  ranking 

5.  Technology  and  manufacturing  readiness  level  (TRL  and  MRL)  assessments  and  metrics 

6.  Requirements  risk  assessment  metrics 

7.  Critical  risk  management  of  software  issues,  e.g.,  complexity,  size,  processing  speed,  throughput, 

schedules,  COTS  availability,  legacy  reuse  suitability,  and  software  development  processes  and 

tools 

8.  A  comprehensive  risk  assessment  for  the  follow-on  phases 

9.  TRL  and  MRL  assessments,  and  metrics 
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10.  Thresholds  and  appropriate  action  plans  for  cases  when  thresholds  are  breached 

1 1 .  The  risk  mitigation  strategies  are  feasible,  and  alternative  courses  of  action  are  identified 

B.  A  demonstrated  degree  of  RM&M  in  all  aspects  of  the  system,  segment,  interface,  and  program  exists 
to  allow  with  an  acceptable  risk  to  proceed  to  CDR 
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Appendix  E  -  Critical  Design  Review  (CDR) 


50.  Critical  Design  Review  (CDR) 

The  CDR  is  a  multidisciplinary  technical  product.  SE  processes  are  normally  held  during  new  Systems 
Development  &  Demonstration  (SDD)  to  assess  whether  the  system  and  configuration  item  or  an 
aggregation  and  functionally  related  group  of  CIs  in  the  specification  tree  under  review  is  sufficiently 
mature  to  proceed  into  fabrication,  demonstration,  and  test,  e.g.: 

a.  Can  meet  the  stated  performance  requirements  within  program  cost,  budget,  schedule,  risk,  and 
other  user  constraints 

b.  The  flowdown  of  requirements  from  the  functional  baseline  to  the  lowest-level  Cl  for  each  end 
item  in  the  specification  tree  is  complete  and  captured  in  each  configuration  item  detailed 
design 

c.  The  system  final  design  is  captured  in  product  specifications  for  each  configuration  item  in  the 
system  baseline 

d.  Each  Cl  in  the  product  baseline  has  been  captured  in  the  detailed  product  specification  and 
design  documentation 

e.  The  Cl  specifications  and  associated  drawings  for  hardware  are  sufficiently  mature  to  enable 
the  fabrication  of  configuration  items 

f.  The  software  architectural  and  detailed  design  for  all  software  items  under  review  are  complete 
to  the  extent  specified  in  the  SDP,  based  on  the  selected  life  cycle  model(s) 

For  complex  systems,  a  series  of  CDRs  for  each  subsystem  or  configuration  item  are  conducted,  leading 
to  an  overall  system  CDR.  When  individual  reviews  have  been  conducted,  the  emphasis  of  the  overall 
system  CDR  shall  focus  on  configuration  item  functional  and  physical  interface  design,  as  well  as  overall 
system  detailed  design  requirements.  The  CDR  determines  whether  the  hardware,  human,  and  software 
(to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle  model(s))  final  detailed  designs  are 
complete,  and  whether  the  Integrated  Product  Team  is  prepared  to  start  system  fabrication,  demonstration, 
and  test. 

Each  CDR  shall  be  tailored  for  the  review  of  the  technical  scope  and  risk  of  the  system,  segment, 
subsystem.  Cl,  and  used  to  update  the  Systems  Engineering  Plan. 

50.1  General 

A  CDR  shall  be  conducted  when  the  “build-to”  baseline  has  been  achieved,  allowing  the  final  deliverable 
hardware  EI(s)  production  to  proceed.  A  rule  of  thumb  is  that  75%  to  90%  of  the  manufacturing  and 
hardware,  build-to  drawings,  and  associated  instructions  are  complete,  and  that  100%  of  all  critical 
component  (e.g.,  critical  safety  items  and  critical  application  items)  drawings  are  complete. 

System  CDR  shall  be  conducted  prior  to  fabrication,  production,  and  coding  release  to  assess  that: 

a.  The  detailed  design  solutions,  as  reflected  in  the  Hardware  Product  Specification,  Interface 
Design  Document(s),  IDD(s),  and  engineering  drawings  satisfy  requirements  established  by  the 
System  Specification 

b.  The  overall  design  and  manufacturing  risks  associated  with  each  configuration  item  are 
manageable  within  program  cost  and  schedule 

c.  All  design  items  have  advanced  as  a  minimum  to  technology  readiness  level  6  (TRL  6)  and 
manufacturing  readiness  level  6  (MRL  6) 
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For  software,  the  CDR  assesses  the  state  of  the  software  and  the  software  risk  for  all  software  in  the 
El  under  review.  Readiness  to  proceed  to  CDR  for  software  is  determined  by: 

a.  The  results  of  the  SARs  (i.e.,  satisfaction  of  the  SAR  “Acceptance  Criteria”  per  Appendix  C) 
held  prior  to  the  CDR  for  the  software  under  review 

b.  Completion  of  detailed  design  for  all  software  builds  for  which  the  SDP  specifies  such 
completion  by  CDR,  based  on  the  selected  life  cycle  model(s) 

50.2  Purpose 

The  purpose  of  the  CDR  is  to  address  and  resolve  critical  detailed  designs  issues,  e.g.: 

a.  Determine  whether  the  design  details  correctly  and  completely  implement  all  system 
requirements  allocated  to  the  subsystem,  and  whether  the  traceability  of  final  subsystem 
requirements  to  final  system  detailed  design  is  maintained 

b.  Verify  that  the  finding  by  peer  reviews  on  requirements  and  final  detailed  design 
documentation  have  been  captured  and  implemented  in  the  detailed  design 

c.  Determine  that  the  detailed  design  of  the  Cl  under  review  satisfies  the  performance  and 
engineering  specialty  requirements  of  the  Cl  development  specifications 

d.  Establish  the  detailed  design  compatibility  among  the  configuration  item  and  other  items  of 
equipment,  facilities,  computer  software,  and  personnel 

e.  Assess  producibility  and  Cl  risk  areas  (on  a  technical,  cost,  and  schedule  basis) 

f.  Review  the  critical  and  long-lead  HWCI  product  specifications  and  review  of  the  software 
detailed  design  for  critical  software  functionality 

g.  Determine  the  acceptability  of  the  detailed  design,  performance,  and  test  characteristics  of  the 
HWCI  and  SWCI  design  solutions,  e.g.: 

(1)  Determine  the  adequacy  of  the  operation  and  support  documents 

(2)  Review  any  outstanding  and  unresolved  deviations,  waivers,  and  deferrals 

(3)  Updated  design  information,  as  applicable 

h.  Assess  results  obtained  during  in-house  testing,  including  problems  encountered  and  solutions 
implemented  or  proposed 

i.  Assess  the  results  of  the  producibility  analyses  conducted  on  system  hardware 

j.  Validate  that  the  latest  estimates  of  cost  (development,  production,  and  support)  are  consistent 
with  the  detailed  design 

k.  Confirm  that  the  results  of  peer  reviews  of  subsystem  requirements,  subsystem  detailed  design, 
and  plans  for  testing  form  a  satisfactory  basis  for  proceeding  into  system  fabrication, 
demonstration,  and  test 

50.3  Objective 

The  objective  of  the  CDR  is  for  the  Contractor  and  the  Acquisition  agency  to  review  and  assess: 

a.  The  status  of  any  changes  to  the  functional  baseline,  architecture,  and  allocated  baseline  since 
they  were  established 

b.  The  design  baseline  for  each  configuration  item,  including  the  completeness  and  compatibility 
of  interfaces  between  the  items  and  between  the  items  and  other  systems,  facilities,  and 
personnel 

c.  The  basis  for  each  element  in  the  design  baseline  in  terms  of  requirements  and  objective, 
comprehensive,  quantitative  design  trades 
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d.  The  balance  between  performance,  cost,  schedule,  and  risk  for  each  element  in  the  selected 
design  baseline 

e.  The  two-way  traceability  from  the  source  of  the  functional  and  allocation  baselines  to  the 
design  baseline  and  back 

f.  That  the  verification  that  the  design  baseline  can  meet  the  contract  requirements 

g.  That  the  subsystem  detailed  designs  are  evaluated  to  determine: 

(1)  Whether  they  correctly  and  completely  implement  all  system  requirements  allocated  to  the 
subsystem 

(2)  Whether  the  traceability  of  final  subsystem  requirements  to  final  system  detailed  design  is 
maintained 

h.  The  results  of  peer  reviews  on  requirements  and  final  detailed  design  documentation  are  folded 
into  the  latest  estimates  of  cost  (development,  production,  and  support)  and  are  consistent  with 
the  detailed  design 

i.  The  plans  for  testing  form  a  satisfactory  basis  for  proceeding  into  system  fabrication, 
demonstration,  and  test 

Completion  of  the  CDR  shall  provide: 

a.  An  established  system  product  baseline 

b.  An  updated  risk  assessment  for  System  Development  and  Demonstration 

c.  Validation  that  the  contractor’s  system-allocated  baseline  is  consistent  with  the  updated  cost 
analysis  requirements  description  (CARD) 

d.  An  updated  program  development  schedule  including  fabrication,  test,  and  software 
development  critical  path  drivers 

e.  An  approved  SWCI  with  updates  applicable  to  this  phase 
Additionally,  a  review  shall  be  conducted  on  each  prototype  to: 

a.  Evaluate  the  progress,  technical  adequacy,  and  risk  resolution  of  the  detailed  design 

b.  Determine  its  alignment  with  the  evolving  functional  architecture  and  allocated  baseline, 
including  compatibility  of  the  physical  and  functional  interfaces  among  the  item  and  other 
items,  systems,  facilities,  and  personnel.  Note:  a  configuration  item  may  consist  of  hardware 
and  software  elements,  and  include  items  such  as  an  airframe. 

50.4  CDR  “Acceptance  Criteria” 

At  CDR  all  major  systems  engineering  management  elements  and  activities  that  are  program  risk  drivers 
are  considered.  The  intent  of  the  CDR  is  to  ascertain  that: 

a.  The  allocation  baseline  for  each  Cl  has  been  confirmed 

b.  The  physical  (a.k.a.  product)  baseline  has  been  approved 

c.  A  design  release  baseline  has  been  established  and  baselined.  For  software,  these  baselines  are 
complete  to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle  model(s) 

In  preparation  for  and  scheduling  of  a  CDR,  the  contractor  shall  demonstrate  to  the  satisfaction  of  the 
contracting  agency  that: 

a.  All  requirements  criteria  elements  are  addressed 

b.  All  applicable  engineering  activities  have  been  properly  conducted  in  support  of  the  criterion 

c.  Each  criterion  shall  be  deemed  successfully  accomplished 

d.  A  viable  technical  and  program  risk  management  strategy  is  in  place 
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e.  Effective  and  efficient  technical  progress  is  made  towards  meeting  all  cost,  schedule,  and 
technical  performance  requirements 

f.  CDR  objectives  are  met  and  the  data  is  available  for  review  and  reside  in  the  decision  database 
The  CDR  “Acceptance  Criteria”  shall  be  organized  under  the  following  five  major  categories: 

1 .  Systems  Engineering  and  Architecture  Development  (50.4. 1) 

2.  System,  Segment,  and  Subsystem  Design  (50.4.2) 

3.  System  Verification  and  Validation  (50.4.3) 

4.  Engineering  Disciplines  and  Specialty  Engineering  (50.4.4) 

5.  Integrated  Technical  Risk  and  Mitigation  (50.4.5) 

This  review  shall  serve  as  objective  evidence  of  the  contractor’s  technical  effort  that  supports  the 
basic  and  agreed-to  CDR  “Acceptance  Criteria”,  e.g.: 

a)  Does  the  status  of  the  technical  effort  and  design  indicate  operational  test  success 
(operationally  suitable  and  effective)? 

b)  Does  the  detailed  design,  as  disclosed,  satisfy  the  Capability  Development  Document  or  any 
available  draft  Capability  Production  Document? 

c)  Has  the  system  product  baseline  been  established  and  documented  to  enable  hardware 
fabrication  and  software  development  to  proceed  with  proper  configuration  management?  That 
is,  is  the  software  product  baseline  complete  to  the  extent  specified  in  the  SDP  based  on  the 
selected  life  cycle  model(s)? 

d)  Has  the  detailed  design  satisfied  Human  Systems  Integration  (HSI)  requirements? 

e)  Are  adequate  processes  and  metrics  in  place  for  the  program  to  succeed? 

f)  Are  the  risks  known  and  manageable  for  developmental  testing  and  operational  testing? 

g)  Is  the  program  schedule  executable  (technical  and  cost  risks)? 

h)  Is  the  program  properly  staffed? 

i)  Is  the  program  executable  with  the  existing  budget  and  the  approved  product  baseline? 

j)  Is  the  detailed  design  producible  within  the  production  budget? 

k)  Are  Critical  Safety  Items  and  Critical  Application  Items  identified? 

l)  Does  the  updated  cost  estimate  fit  within  the  existing  budget? 

m)  Are  the  schedules  for  completion  of  software  development  consistent  with  status  of  the 
software  design  at  the  time  of  the  CDR? 

n)  Have  key  product  characteristics  having  the  most  impact  on  system  performance,  assembly, 
cost,  reliability,  or  safety  been  identified? 

o)  Have  the  critical  manufacturing  processes  that  impact  the  key  characteristics  been  identified 
and  their  capability  to  meet  design  tolerances  determined? 

p)  Have  process  control  plans  been  developed  for  critical  manufacturing  processes? 

q)  Have  all  IMP  and  IMS  tasks  associated  with  this  review  been  successfully  closed? 

The  program  manager  shall  conduct  the  CDR  when  the  hardware  “build-to”  baseline  has  been  achieved, 
allowing  production  to  proceed. 
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50.4.1  Systems  Engineering  and  Architecture  Development 

Evidence  of  Systems  Engineering  and  Architecture  Development  requirements  maturity  criteria  examples 

at  CDR: 

A.  The  system,  segment,  subsystem,  and  component  allocated  and  physical  requirements  for  each  Cl  are 

complete,  feasible,  verifiable,  and  clearly  stated. 

1 .  Critical  System  Design  is  correlated  with  and  reflected  in  the  Allocated  and  Physical  Baselines, 
e.g.,  critical  design  of  the  system,  segment,  subsystem,  and  component  correlated  with  the  system 
architecture  views  and  descriptions  is  traceable  to  all  baselines  and  maintained  under 
configuration  control 

2.  End-to-end  processing  capability  of  the  system,  segment,  subsystem,  and  hardware  component 
architectures  (including  timelines  and  capacities)  for  production,  integration,  operations, 
maintenance,  and  training  is  verified  and  baselined,  e.g.: 

a.  Critical  design  of  the  system,  segments,  subsystems,  and  components  considers  the  physical 
hierarchy  extended  to  identify  all  additional  products  necessary  to  manufacture,  verify, 
integrate,  deploy,  train,  operate,  support,  sustain,  and  dispose  of  the  system,  its  constituent 
elements,  and  components  over  its  life  cycle 

b.  Final  technical  design  includes  all  build-to  (including  drawings,  and  processing  and  assembly 
instructions),  buy-to,  or  verify-to  (including  design  qualification  and  delivery  acceptance 
verifications  as  well  as  tests  for  workmanship);  integrate-to,  deploy-to  (including 
verifications  of  operational  readiness),  train-to,  operate-to  (including  tech  orders  and 
operating  instructions),  support  and  sustain-to  (including  maintenance  and  support  tests), 
and/or  dispose-to  requirements  for  each  product  (except  government  property)  satisfying 
requirements,  functional  allocated  and  physical  baselines 

3.  Final  design  data  (e.g.,  drawings,  specifications,  etc.)  for  the  system,  segments,  subsystems,  and 
components  is  completed  down  to  their  constituent  element  and  unit  levels,  e.g.,  separable 
documentation  exists  for  each  element  and  component  of  the  physical  hierarchy,  and  for  each 
additional  system  product  or  integrated  grouping  of  products  that  is  separately  manufactured, 
procured,  authored  (in  the  case  of  manuals  and  other  written  and  drawn  products),  verified, 
integrated,  deployed,  trained  for,  operated,  supported,  or  disposed  of,  and  any  others  as  required 
by  the  customer 

4.  The  source  requirement  and  tradeoff  or  other  basis  for  each  design  solution  or  other  element  of 
the  design  release  baseline  is  captured  in  the  decision  database  and  linked  to  the  element 

B.  System  Requirements  Allocation  is  completed  and  verified  for  all  CIs 

1.  Requirements  flowdown  and  derivation  from  subsystems  down  to  component  elements  and  unit 
levels  are  complete  and  traceable  (no  TBDs,  TBSs,  TBRs,  or  deferrals  are  identified) 

2.  Design-to  and  off-the-shelf  (OTS)  specifications  are  completed  and  validated  by  production, 
verification,  and  operations  organizations  and  by  specialty  engineering  groups 

3.  Final  functional  flow  block  diagrams  (FFBDs)  are  completed  down  to  the  hardware  component, 
element,  and  unit  level  for  the  system,  segments,  subsystems  and  all  interfaces  (internal  and 
external)  demonstrating  flowdown  and  traceability  between  higher-  and  lower-level  allocated 
requirements 

4.  The  system  component,  element,  and  unit  design  specifications  are  under  configuration 
management  without  any  major  TBDs  or  open  items 

5.  Final  production  requirements  are  developed  and  documented 
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C.  Physical  Baseline  is  established  and  traceable  to  the  approved  Mission,  System  Functional,  and 
Allocation  Baselines. 

1.  Physical  baseline  includes  all  allocated  and  derived  design-to  requirements  and  design  constraints 
for  each  product  in  the  physical  hierarchy 

2.  System  physical  requirements  are  allocated  to  all  system  segments,  subsystems,  and  hardware 
components 

3.  System,  segment,  subsystem,  hardware,  and  component- level  physical  analyses  are  completed, 
down  to  the  element  and  unit  level,  and  are  traceable  to  accepted  trade  results  (design  choices) 

4.  Interoperability  physical  requirements  are  allocated  to  all  system,  segment,  subsystem,  hardware 
component,  and  external  interface  critical  designs 

5.  Physical  baseline  complies  with  CONOPS  and  the  contractor’s  OpsCon 

D.  Baseline  (BL) 

1.  Life  cycle  cost  analysis  results  include  sensitivity  of  physical  parameters  to  cost 

2.  Cost  models  representing  final  approved  program  development,  operational,  and  sustainment 
costs  are  baselined  and  include  cost  impacts  to  other  systems 

3.  Results  from  life  cycle  cost  and  systems  performance  trade  studies  are  maintained  and  rationale 
for  changes  identified 

E.  System  integration  and  verification  physical  requirements  are  allotted  down  to  the  component, 
element,  and  unit  level 

1.  Component,  element,  and  unit-level  verification  planning  is  completed  with  rationale  for 
verification  objectives,  types,  levels  and  sequence  of  verification,  venues,  and  verification  data  to 
be  collected 

2.  Component,  element,  and  unit-level  integration  and  test  planning  is  completed  with  rationale  for 
test  objectives,  type,  level  and  sequence  of  testing,  test  venues,  and  test  data  to  be  derived 

3.  Processes  and  procedures  are  completed  for  system  integration  and  verification,  e.g.,  final 
processes  and  procedures  are  verified  and  baselined  for  segment,  subsystem,  and  component 
integration  and  verification  down  to  the  element  and  unit  levels. 

4.  Segment,  subsystem,  and  component-level  cross-reference  requirements  are  completed  and 
baselined  down  to  the  element  and  unit  levels. 

F.  System,  Segment,  Subsystem,  and  Component-level  interfaces  are  completed 

1.  Final  internal  interfaces  design  (component-to-component,  unit-to-unit)  is  completed 

2.  Final  external  interfaces  design  (system,  system  of  systems,  and  family  of  systems)  is  completed 

G.  Physical  descriptions  and  parameters  are  completed  for  each  HWCI  and  SWCI 

1.  Physical  baseline  for  HWCIs  and  SWCIs  is  traceable  to  the  requirements,  functional,  and 
allocation  baselines 

2.  Physical  baseline  for  all  HWCIs  and  SWCIs  is  under  configuration  control,  e.g.,  all  changes  to 
the  requirements,  functional,  and  allocation  baselines  are  identified,  tracked,  and  documented 

H.  System  performance  (design)  specification  is  traceable  to  the  allocated  and  physical  baselines,  e.g.,  all 
specifications  down  to  the  component,  element,  and  unit  are  developed  and  are  traceable  to  the 
system  performance  specification 

I.  Design  Release  Baseline  is  defined  for  the  critical  design  and  is  traceable  to  the  functional,  allocated, 
and  physical  baselines 

1.  Adequate  information  exists  (e.g.,  design  drawings,  design  specifications,  test  and  analysis  data) 
to  support  a  final  design  release  baseline 
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2.  The  design  release  baseline  was  developed  iteratively,  based  on  tradeoffs  and  planning, 
monitoring,  decisions,  and  control;  adequate  tradeoffs  were  performed  to  support  each  final 
design  selection 

3.  The  physical  (a.k.a.  product)  configuration  baseline  is  used  for  building,  or  buying,  and  then 
integrating  the  development  products  to  verify  that  the  requirements  in  the  allocated  baseline 
were  achieved  and  to  validate  that  the  integrated  system  fulfills  the  users’  capabilities  needed 

J.  Development  of  long-lead  production  specifications  is  completed  and  baselined 

1.  Production  specifications  are  traceable  to  the  allocated  and  physical  baselines 

2.  Critical  production  (shop)  drawings,  Fabrication  and  Assembly,  Integration  and  Test  (F/A,  I&T) 
processes  and  procedures  are  baselined  and  put  under  configuration  control 

K.  The  nondevelopmental  software  and  hardware  items  (NDI)  (e.g.,  COTS,  GOTS,  and  reuse  software) 
are  reviewed  to  ensure  that  they  do  not  add  additional  constraints  on  the  system 

50.4.2  System,  Segment,  and  Subsystem  Design 

Evidence  of  System,  Segment,  and  Subsystem  Design  Concepts  maturity  criteria  examples  at  CDR: 

A.  System,  segment,  subsystem  and  component  critical  design  (down  to  the  element  and  unit  levels)  is 
completed 

1.  Critical  design  demonstrates  traceability  among  all  considerations:  allocated  and  physical 
requirements,  engineering  trade  study  results,  technology  selections,  and  technical,  programmatic, 
schedule,  and  cost  risks,  e.g.: 

a.  The  adequacy  of  the  critical  design  has  been  demonstrated  using  ongoing  engineering 
analyses,  considering  all  relevant  specialty  engineering  disciplines 

b.  Engineering  analyses  adequately  support  the  physical  (a.k.a.  product)  requirements  for 
hardware  and  software  configuration  items  down  to  the  component,  element,  and  unit  levels. 

c.  Engineering  analysis  results  adequately  demonstrated  the  readiness  of  the  design  to  proceed 
to  production 

d.  Engineering  analysis,  modeling,  and  simulation  results  supporting  critical  design  capabilities 
and  solutions  are  verified  and  baselined 

2.  Demonstrate  that  the  critical  design  is  traceable  to  and  correlated  with  all  critical  allocated  and 
physical  requirements 

3.  Physical  requirements  derived  from  the  allocation  baseline  for  segments  and  subsystems  represent 
a  complete  and  optimal  synthesis  of  the  component-,  element-,  and  unit-level  requirements  design 

4.  Appropriate  margins,  allowances,  and  contingencies  are  established  at  the  segment,  subsystem, 
and  component  levels  down  to  their  elements  and  units 

5.  Design  development  planning  is  executed  and  tracked,  e.g.,  critical  design  drawings  put  under 
configuration  control  (at  PDR)  are  maintained,  and  changes  are  documented  with  supportive 
rationale 

6.  Final  electrical,  mechanical,  and  functional  performance  schematics  are  available,  including 
functional  flow  block  diagrams  (FFBDs)  for  inter-  and  intra-segments  and  subsystems  down  to 
their  component  elements  and  unit  levels 

7.  Ground  Support  Equipment  (including  common,  peculiar,  flight,  and  nonflight  test  support 
equipment  and  tooling)  selections  are  baselined  and  initial  designs  are  completed,  e.g.: 

a.  GSE  make-or-buy  decisions  are  baselined 

b.  Initial  GSE  Hardware  Allocation  Listing  (HAL)  is  completed 
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8.  Government- furnished  equipment  (GFE)  and  ancillary  test  hardware  are  verified  and  baselined 
for  their  intended  use 

B.  C4I  physical  and  software  allocations  (to  the  extent  specified  in  the  SDP  based  on  the  selected  life 
cycle  model(s)),  are  incoiporated  into  the  critical  design  across  segments,  subsystems,  and  hardware 
components,  in  addition  to  system  interrelationships  and  interdependency: 

1.  Physical  allocations  include  battle  management  and  information  technology  (IT)  needs, 
dependencies,  and  interfaces  between  system  segments,  subsystems,  and  the  system,  and  system 
of  systems  and  family  of  systems 

2.  Physical  allocations  ensure  C4I  interoperability,  interconnectivity,  supportability, 
synchronization,  and  sufficiency 

C.  Threat  scenarios  and  threat  environment  parameters  correlated  with  the  critical  design,  e.g.,  the  threat 
scenario,  the  operational  and  the  environmental  allocations  incoiporated  into  the  critical  design  are 
traceable  to  all  segments,  subsystems,  and  components  down  to  their  elements  and  unit  levels 

D.  Environmental  (e.g.,  natural,  thermal,  humidity,  transport)  parameters  correlated  to  the  critical  design, 
e.g.,  the  environmental  allocations  are  incorporated  into  the  critical  design  and  are  traceable  to  all 
segments,  subsystems,  and  components  down  to  their  element  and  unit  levels 

E.  Reliability,  availability,  maintainability,  and  testability  (RAM&T)  allocated  requirements  are 
incorporated  into  the  critical  design,  e.g.,  RAM&T  allocations  are  traceable  to  segments,  subsystems 
and  components  down  to  their  elements  and  unit  levels  for  hardware  and  to  the  SWCI  for  software 

F.  System  operational  sustainment  key  performance  parameters  are  incoiporated  into  the  critical  design, 
including  all  major  system  and  program  requirements,  and  the  updated  LCC  and  CAIV  modeling  and 
analysis  studies  presented  at  the  PDR,  e.g. : 

1.  LCC  and  CAIV  modeling  and  analyses  are  applied  and  correlated  for  each  HW  and  SW  design 

2.  They  accurately  depict  projected  program  development,  operational  and  sustainment  costs,  as 
well  as  projected  cost  impacts  to  other  “external”  systems 

G.  LCC  sustainment  model  is  correlated  with  the  critical  design 

H.  Risk  mitigation  solutions  in  the  system  risk  model  are  traceable  to  and  correlated  with  the  critical 
design 

I.  Ongoing  Industrial  Base  assessment  results  are  correlated  with  the  critical  design;  new  risk  areas  (not 
identified  at  PDR)  are  prioritized  and  the  mitigation  process(es)  are  defined,  including  resources  and 
schedule  requirements 

1.  IB  assessment  data  (e.g.,  DMSMS,  parts  obsolescence)  are  correlated  with  identified  and  implicit 
design  and  production  risk  areas 

2.  Mitigation  strategies  are  planned  and  implemented,  including  resources  and  schedule 
requirements 

J.  Key  allocated  performance  requirements  are  traceable  to  the  critical  system  design  for  all  major 
subsystems  and  components 

1.  All  major  subsystem  and  component  allocations  are  incorporated  into  the  critical  design 

2.  Key  parameters  and  information  (developed  and  assessed  at  PDR)  are  implemented  for  each 
major  subsystem  and  component  critical  design,  e.g.: 

a.  Major  performance  parameters  are  incorporated 

b.  Critical  technologies  are  under  development 

c.  Critical  design  and  manufacturing  requirements  and  challenges  (identified  at  SFR)  are 
correlated  with  critical  design 
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Note:  The  following  examples  are  intended  to  provide  clarification  of  the  types  of  data  and  level  of  detail 
expected  to  be  addressed  at  CDR.  It  is  intended  that  the  contractor  will  identify  those  subsystems  and 
components  applicable  to  the  type  of  system  being  developed  and  the  appropriate  criteria  for  each 
subsystem  and  component  necessary  to  effectively  evaluate  and  assess  the  critical  system  design,  its 
technical,  cost,  and  schedule  parameters,  and  demonstrate  that  the  design  has  incorporated  recovery 
modes  for  all  failure  modes  identified  at  PDR,  e.g.: 

=>  For  Electrical  Power: 

•  Electrical  Power  Distribution  System  (EPDS)  performance  requirements,  characteristics,  and 
operational  criteria  are  defined,  including  initial  power  budgets,  total  power  demand  with 
allowable  margins,  and  modes  of  operation  (frequency  and  duration) 

•  Selection  and  evaluation  of  the  type(s)  of  power  supply  sources  are  being  considered,  along  with 
their  specific  technology  and  topology 

•  Battery  (or  energy  storage)  power  requirements  are  identified  and  modes  of  operation  defined 
(frequency  and  duration) 

•  Battery  life  requirements  (BOL  and  EOL)  and  other  unique  requirements  that  may  impact  battery 
selection  or  design  are  defined 

•  Battery  cell  technologies  are  identified  and  battery  architectures  defined 
=^>  For  Software: 

•  The  “Acceptance  Criteria”  for  software  detailed  in  Appendix  C,  Sections  30.4,  e.g.,  SAR 
“Acceptance  Criteria”  (paragraphs  A,  B,  C,  D,  F,  G,  H,  I,  and  J)  are  satisfied  to  the  extent 
specified  in  the  SDP  based  on  the  selected  life  cycle  model(s). 

50.4.3  System,  Segment,  and  Subsystem  Verification  and  Validation 

Evidence  of  System,  Segment,  and  Subsystem  design  concepts  verification  and  validation  (V&V) 
requirements  maturity  criteria  examples  at  CDR: 

A.  System,  Segment,  Subsystem,  and  Component  V&V  approaches  developed  for  the  critical  design 

1.  Critical  design  demonstrates  that  major  system-,  segment-,  subsystem-,  and  component-allocated 
requirements  can  be  verified  and  validated,  e.g.: 

a.  V&V  approaches  are  developed  for  the  critical  design  address  system  of  systems,  system, 
segment/subsystem,  and  component  down  to  their  element  and  unit  levels 

b.  V&V  approaches  include  analytical,  modeling,  and  simulation  and  testing  processes  and 
procedures  for  critical  design 

c.  V&V  processes  and  procedures  address  new  technology,  verification,  and  qualification 
technical  practices,  system-level  demonstrations  and  tests,  support  required  from  external 
organizations  and/or  facilities,  and  resource  requirements  for  the  critical  design 

d.  V&V  processes  and  procedures  for  the  critical  design  based  on  proven,  referenced  practices 

e.  Updated  subsystem  and  component  VCRMs  are  complete  and  consistent  with  system-  and 
segment-allocated  requirements  and  intemal/extemal  interface  allocated  requirements,  e.g. : 

(1)  Components  element  and  unit  VCRMs  are  traceable  to  the  system/segment/subsystem 
VCRMs 

(2)  The  updated  segment/subsystem/component  VCRMs  are  baselined  and  under 
configuration  management 

(3)  V&V  methods  in  the  system  and  segment/subsystem/component  VCRMs  are  adequate  to 
verify  the  system  and  its  segments/subsystems/components  down  to  their  element/unit 
levels 
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B.  System  operational  functions  and  environments  for  the  critical  design  are  traceable  to  the  contractor’s 
operations  concept  (OpsCon),  and  the  allocated  and  physical  baselines. 

1.  Demonstrate  that  the  system  V&V  test  environment  allocations  are  traceable  to  the  system 
performance  specification  for  the  critical  design 

2.  Demonstrate  that  the  critical  design  is  correlated  with  and  traceable  to  all  critical  allocated  and 
physical  environmental  parameters,  V&V  approaches,  and  processes 

C.  DT&E  elements  are  correlated  with  the  critical  design 

D.  OT&E  allocated  and  physical  requirements  are  incorporated  into  the  critical  design 

E.  Test  bed(s)  and  test  facilities  are  chosen  based  on  the  critical  design  are  deemed  adequate  to  perform 
system,  segment/subsystem,  and  interface  requirements  verification,  e.g.,  critical  hardware  and 
software  items  procurement  and  scheduling  are  complete  and  in  place  as  V&V  resources  (e.g., 
simulators,  test  beds,  test  facilities) 

F.  Test  requirements  and  test  data  collected  to  date  for  the  critical  design  are  traceable  to  operational 
requirements  via  specifications  and  V&V  cross-reference  matrices  (VCRMs),  e.g.,  use  of  comparative 
test  data  to  anchor  representative  system/segment/subsystem  models  and  simulations  down  to  their 
element  and  unit  levels  to  real-world  environments  and  allocated  and  physical  requirements  are 
demonstrated 

G.  V&V  risk  approaches,  processes,  and  procedures  are  developed  for  the  critical  design. 

FI.  V&V  test  deficiencies,  including  those  based  on  technology  deficiencies,  are  established  at  PDR,  and 
correlated  with  the  critical  design  and  the  impact  assessed 

1.  Risk  mitigation  approaches  are  developed  and  integrated  into  the  system  risk  model,  including 
resource  requirements,  which  are  correlated  with  the  critical  design 

50.4.4  Engineering  Disciplines/Specialty  Engineering 

Evidence  of  Engineering  Discipline/Specialty  Engineering  identification  and  assessment  maturity  criteria 
(categories  listed  in  A  through  R  below)  at  CDR  in  terms  of,  e.g.: 

1 .  Key  performance  requirements 

2.  Key  performance  parameters 

3.  Use  of  heritage  systems/components/technology 

4.  Use  of  new  designs 

A.  Parts,  Materials,  and  Processes  (PM&P) 

1.  PM&P  allocated  and  physical  requirements  are  incorporated  into  the  critical  design 

2.  Environments  and  environmental  parameters  impacting  parts  performance  are  incorporated  into 
the  critical  design 

3.  Parts  engineering  design  analyses  are  completed  for  the  critical  design  addressing  risk 
assessments,  technologies,  sources  of  supply,  and  the  common  quality  levels  (i.e.,  reliability)  of 
the  parts,  e.g.,  results  of  critical  design  analyses  used  to  develop  final  critical  parts  and  long-lead 
items  list  are  complete 

B.  Test  and  Evaluation  (T&E) 

1.  Final  T&E  planning  is  traceable  to  the  critical  design  correlating  all  test  objectives,  test 
environments,  and  test  resources  to  allocated  requirements 

2.  T&E  approaches  (selected  at  PDR)  are  verified  and  correlated  with  the  critical  design,  e.g.: 

a.  Demonstrate  that  the  baselined  test  processes  and  procedures  developed  at  PDR  can  verify 
the  system,  segments,  and  subsystems  allocated  and  physical  requirements  and  interfaces 
down  to  their  component  elements  and  units 
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b.  Baselined  T&E  processes  and  procedures  capture  the  characteristics,  effectivity(s),  and 
margins  for  each  particular  test  item  down  to  their  elements  and  unit  levels 

3.  Test/verification  data  gathering,  reduction,  and  analysis  processes  for  the  critical  design  are 
verified  and  baselined,  including  test  environment(s),  operations,  and  procedures  to  be  performed, 
data  acquisition  requirements,  documentation,  methods  of  analysis,  and  pass-fail  (i.e.,  success) 
criteria 

C.  Survivability  and  Vulnerability 

1.  Demonstrate  that  the  critical  design  captures  the  survivability  and  vulnerability  threat  allocations 
incorporated  into  the  preliminary  design  for  all  categories  of  expected  threats,  threat 
environments,  and  their  likelihood  of  occurrence 

2.  Demonstrate  that  the  system/threat  interaction  analyses  that  established  and  baselined  threat 
margins  at  PDR  are  still  adequate  and  complete  for  the  critical  design 

3.  Survivability  design  solutions  are  correlated  with  and  incorporated  into  the  critical  design  shown 
to  mitigate  each  known  threat 

D.  Environmental  Safety  and  Occupational  Elealth  (ES&OH) 

1.  Life  cycle  environmental  allocations  are  incorporated  into  the  critical  design 

2.  Data  compiled  for  CDR  Programmatic  ES&OH  Evaluation  (PESHE)  compliance  objectives  are 
correlated  with  the  critical  design,  including  an  assessment  of  internal  and  external  operational 
environments 

3.  Elazardous  materials  management  and  pollution  prevention  processes  and  procedures  are  verified 
and  baselined  to  the  critical  design 

4.  Critical  human  safety  and  health  factors  are  baselined  and  incorporated  in  the  critical  design 

E.  Mass  Properties 

1.  Mass  properties  margins  (average  or  complex)  established  for  CDR  are  correlated  to  the  critical 
design,  including  allowable  growth  allocations  and  metrics 

2.  Calculated  weight  growth,  center  of  gravity,  and  moments  of  inertia  parameters  are  allocated  to 
the  critical  design 

F.  System  Security  Engineering  (SSE),  Communications  Security  (COMSEC),  Information  Assurance 

(IA),  and  Program  Protection  (PP): 

1.  Requirements  are  incorporated  into  the  critical  design  IAW  DoD/AF  policies,  directives,  and 
system  specifications,  e.g.,  program  protection  planning  is  complete  and  ready  for  Milestone 
Decision  Authority  (MDA)  approval 

2.  System  security  concept  and  specification  implemented  for  the  critical  design  include  a  finalized 
baseline  security  test  and  evaluation  processes  and  procedures 

3.  Threat,  vulnerability,  risks  assessments,  and  baselined  protection  countermeasures  implementing 
the  trusted  facilities  are  complete 

4.  Information  Assurance  controls  included  in  the  critical  design  and  certification  and  accreditation 
requirements  are  finalized  following  the  DIACAP 

5.  Program  baseline  costs  for  SSE,  COMSEC,  IA,  and  PP  for  implementation  and  sustainment  of 
the  system  are  updated 

G.  Interoperability 

1.  Allocated  and  physical  system  and  mission  interoperability  requirements  are  incorporated  into  the 
critical  design 
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2.  Allocated  and  physical  requirements  from  new/unique  standards  approved  for  inclusion  in  DISR 
(i.e.,  new  data  formats,  interdependency,  data  exchange  protocols/schemas,  Ethernet  alternatives) 
are  correlated  with  and  incorporated  into  the  critical  design 

3.  Allocated  and  physical  interoperability  requirements  for  all  the  interrelationships  and 
interdependency  are  incorporated  into  the  critical  design 

H.  Reliability  and  Maintainability  (R&M) 

1.  R&M  allocated  and  physical  requirements  are  incorporated  into  the  critical  design 

2.  R&M  analyses  results  are  correlated  with  the  critical  design,  e.g.: 

a.  Approaches  and  processes  developed  for  implementing  Environmental/Thermal  Stress 
Screening  (ESS/TSS)  at  PDR  are  verified  and  baselined  for  the  critical  design 

b.  Packaging,  Handling,  Storage,  and  Transportability  (PHS&T)  environmental  allocated  and 
physical  requirements  in  the  R&M  program  are  incorporated  into  the  critical  design 

3.  Update  the  hardware  FMECA  and  the  RMA&D  Prediction  Analyses  (including  final  Reliability 
Stress  Analysis  -  with  software  if  applicable)  for  final  design,  e.g.: 

a.  Update  critical  items  list  and  single-point  failures  list 

b.  Update  any  safety  issues  and  associated  analyses  as  appropriate 

c.  FMECA  is  to  include  effects  of  design  implementation,  e.g.,  proximity  of  parts,  location  in 
wire  bundles,  etc. 

I.  EMI/EMC 

1 .  Electromagnetic  interference  control  processes  and  procedures  developed  at  PDR  are  verified  and 
baselined  for  the  critical  design 

2.  Internal  and  external  EMI/EMC  allocated  and  physical  requirements  are  incorporated  into  the 
critical  design 

3.  EMI  susceptibility  allocated  and  physical  requirements  and  constraints  are  incorporated  into  the 
critical  design 

4.  Demonstrate  that  the  preliminary  design’s  EMI/EMC  critical  environmental  characteristics  and 
sensitive  elements  are  correlated  to  the  critical  design 

J.  Human  Systems  Integration  (HSI) 

1.  User  interface  hardware/software  allocated  and  physical  requirements  for  operators,  users, 
maintainers,  and  sustainers  are  incorporated  into  the  critical  design 

2.  Usability,  maintainability,  operability  and/or  supportability  physical  requirements  decomposed 
from  system  allocated  requirements  are  incorporated  into  the  critical  design 

3.  Operational  manning,  workload,  and  skill  level  are  allocated,  and  physical  requirements  are 
incorporated  into  the  critical  design 

K.  Manufacturing  and  Producibility 

1.  Demonstrate  that  the  manufacturing  and  producibility  approaches  and  processes  are  developed 
and  correlated  with  the  critical  design 

2.  Producibility  procedures  and  methods  verified  and  baselined  for  the  critical  design  demonstrate 
the  manufacturing  processes  selected  at  PDR  support  the  critical  design 
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L.  Life  Cycle  Logistics 

1.  Supportability  is  allocated  and  physical  requirements  are  incoiporated  into  the  critical  design 

2.  System-level  logistics  elements  are  correlated  with  the  critical  design,  including  design  interface, 
supply  support,  test  equipment,  manpower  and  personnel,  training  and  training  equipment, 
PHS&T,  facilities,  computer  resources,  technical  data,  and  maintenance  planning 

3.  Logistics  Management  Information  (LMI)  is  completed  and  validated  in  support  of  the  Allocation 
and  Physical  baselines  for  the  critical  design 

M.  System  Safety 

1.  System  safety  is  allocated  and  physical  requirements  are  incorporated  into  the  critical  design 

2.  Segment/subsystem  hazard  analyses  are  completed  and  an  updated  list  of  prioritized  safety 
hazards  established  for  the  test,  operation,  and  disposal  of  the  critical  design 

3.  Critical  human  safety  and  health  factors  are  baselined  and  incoiporated  into  the  critical  design 

4.  The  baselined  hazardous  materials  list  (compiled  and  prioritized  at  PDR)  correlated  to  and 
updated  as  required  for  the  critical  design 

N.  Contamination  Control 

1.  Contamination  control  processes  and  procedures  developed  at  PDR  are  verified  and  baselined  for 
the  critical  design 

2.  Material  outgassing  survey  results  (from  PDR)  are  correlated  with  and  updated  as  required  for  the 
critical  design 

O.  Quality  Assurance 

1.  Quality/product  assurance  allocated  and  physical  requirements  correlated  to  and  incoiporated  in 
the  critical  design 

2.  Verification,  inspection,  and  test  processes  and  procedures  developed  at  PDR  are  verified  and 
baselined  for  the  critical  design 

P.  Environmental  Considerations 

1.  Environmental  study  results  (from  PDR)  are  correlated  with  and  updated  as  required  for  the 
critical  design 

2.  Environmental  test  and  evaluation  approaches  and  processes  are  developed  for  the  critical  design 

3.  Reliability-thermal  allocation  requirements  are  incoiporated  into  the  critical  design 

Q.  Software 

1 .  Requirements 

a.  Software  requirements  (including  software  interface  requirements)  have  been  specified  to  the 
level  of  completeness  called  for  in  the  software  development  plan  based  on  the  selected 
software  life  cycle  model 

b.  Software  requirements  (including  software  interface  requirements)  are  correct,  complete, 
consistent,  feasible,  verifiable,  and  clearly  and  unambiguously  stated 

c.  Software  requirements  (including  software  interface  requirements)  are  traced  to  and  fully 
implement  their  parent  requirements 

d.  Software  requirements  include  necessary  requirements  derived  from  the  system  and  software 
architecture,  system  operational  concepts,  trade  studies,  or  design  decisions 
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2.  Each  software  requirement,  including  software  interface  requirements,  has  one  or  more  valid 
verification  methods  and  verification  levels  specified,  and  those  methods  and  levels  are 
sufficient  to  fully  verify  the  requirement 

a.  The  “Acceptance  Criteria”  for  software  detailed  in  Appendix  C,  sections  30.4  SAR 
“Acceptance  Criteria”  (paragraphs  A,  B,  C,  D,  F,  G,  H,  I,  and  J)  are  satisfied  to  the  extent 
specified  in  the  SDP  based  on  the  selected  life  cycle  model(s) 

b.  Software  operational  concepts  include  nominal  and  off-nominal  scenarios  from  a  software 
perspective  (e.g.,  start-up/initialization,  shutdown,  processor  failover,  redundancy 
management,  recovery/restoral)  consistent  with  the  system  and  software  architectures 

c.  Software  operational  concepts  include  information  exchange  with  external  interfacing 
systems 

d.  Software  operational  concepts  include  scenarios  for  operational  workloads 

e.  Software  operational  concepts  are  consistent  with  system  operational  concepts 

3.  Architecture  and  Design 

a.  The  software  architecture  has  been  defined  to  the  level  of  completeness  called  for  in  the 
software  development  plan,  based  on  the  selected  software  life  cycle  model 

b.  The  software  architecture  views,  including  the  physical,  logical,  developmental,  process,  and 
behavioral  (user)  views,  are  correct,  complete,  consistent,  clear,  and  unambiguous 

c.  Nondevelopmental  items  (NDI)  (e.g.,  COTS,  GOTS,  and  reuse  software)  have  been  fully 
integrated  into  the  components  of  the  software  architecture 

d.  The  software  architecture,  including  the  nondevelopmental  items  (NDI)  (e.g.,  COTS,  GOTS, 
and  reuse  software),  will  enable  the  higher-level  requirements  allocated  to  software,  the 
software  requirements,  and  the  software  interface  requirements  to  be  met 

e.  The  design  of  each  software  item  has  been  elaborated  to  the  level  of  software  units,  consistent 
with  the  software  development  plan  and  the  selected  software  life  cycle  model 

f.  The  design  of  each  software  item  is  clear,  correct,  complete,  consistent,  and  unambiguous, 
and  adequately  addresses  the  following: 

(1)  Detailed  design  of  all  external  and  internal  interfaces 

(2)  Detailed  design  of  all  files,  databases,  shared  memory,  etc.,  and  their  storage  and  access 
methods 

(3)  Detailed  design  of  user  interface  screens  and  human/system  interactions 

(4)  Source  for  each  unit  of  the  software  item  (i.e.,  COTS,  unmodified  reuse,  modified 
reuse,  or  newly  developed  code),  and  programming  language(s)  to  be  used 

(5)  Selected  COTS  software  products  and  installation/configuration  design  decisions 

(6)  Detailed  design  of  glue  code  for  integrating  COTS  and  reuse  software  products  with 
each  other  and  with  the  newly  developed  code 

(7)  Detailed  algorithm  designs  for  the  software  units,  including  both  mathematical  and 
procedural  algorithms 

(8)  Detailed  design  of  the  dynamic  structure  of  the  software  items  (e.g.,  processes/tasks, 
flow  of  execution  control,  priorities,  sequencing,  dynamic  creation/deletion  of  process) 

(9)  Detailed  design  of  exception  handling  and  recovery  methods 

(10)  Application  Programming  Interfaces  (APIs)  to  be  used  (both  standardized  APIs  and 
APIs  uniquely  defined  for  this  system) 
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g.  The  design  of  each  software  item  properly  implements  all  applicable  standards  (e.g.,  interface 
standards,  graphical  user  interface  (GUI)  standards) 

h.  Updated  software  architecture  and  design  adequately  address  the  use  of  open  systems 
standards  and  satisfy  all  applicable  interoperability-related  requirements 

i.  Updated  software  architecture  and  design  adequately  address  end-to-end  processing 
(including  timelines  and  capacity)  for  operations,  maintenance,  and  training,  across  elements 
and  external  and  internal  interfaces 

j.  Updated  software  architecture  and  design  adequately  address  operational  database 
management  and  control 

k.  Updates  to  selected  computing  resources  (e.g.,  processors,  cache,  memory,  buses,  and 
networks)  are  appropriately  incoiporated  into  the  updated  system  and  software  architectures, 
and  will  enable  the  allocated  element,  subsystem,  software,  and  interface  requirements  to  be 
met 

l.  Updated  software  architecture  and  detailed  design  meet  appropriate  functional  and 

performance  requirements  for  each  state  and  mode 

m.  Updated  software  architecture  and  detailed  design  adequately  address  requirements  for 
survivability  and  endurability  from  a  computer  hardware  and  software  perspective 

n.  Updated  software  architecture  and  detailed  design  adequately  address  supportability, 

including  fault  management  and  integrated  hardware-software  diagnostics,  fault  detection, 
isolation,  localization,  restoral,  and  repair 

o.  Updated  software  architecture  and  detailed  design  adequately  address  dependability, 

reliability,  maintainability,  and  availability  requirements  allocated  to  the  computer  hardware 
and  software  subsystems 

4.  Engineering  Analysis 

a.  Updated  engineering  analyses,  models,  and/or  simulations  adequately  demonstrate  that  the 
software  architecture  and  detailed  design,  together  with  the  computer  resources  (hardware 
and  software)  that  have  been  selected,  will  meet  the  Key  Performance  Parameters  (KPPs)  and 
driving  requirements 

b.  Updated  reliability,  maintainability,  and  availability  analyses  are  consistent  with  the  software 
architecture  and  detailed  design  and  with  the  computer  resources  (hardware  and  software) 
that  have  been  selected,  and  appropriately  include  the  contribution  of  software 

c.  Updated  safety,  information  assurance,  and  human  systems  integration  analyses  are 
consistent  with  the  software  architecture  and  detailed  design  and  with  the  computer  resources 
(hardware  and  software)  that  have  been  selected,  and  appropriately  include  the  contribution 
of  software 

d.  Updated  engineering  analyses  and  trade  studies  adequately  support  software  architectural  and 
detailed  design  decisions  about  NDI  (reuse,  COTS,  and  GOTS  software  components),  and 
appropriately  consider  the  underlying,  supporting  computer  resources  (hardware  and 
software)  that  have  been  selected 

e.  Updated  human  systems  integration  engineering  analyses  and  trade  studies  (e.g.,  operability, 
operator  workload  analysis)  demonstrate  the  adequacy  of  the  software  architecture  and 
detailed  design  and  the  computer  resources  (hardware  and  software)  that  have  been  selected 
for  the  operators  to  perform  their  required  roles  within  the  required  timelines 

f.  Updated  performance  analysis  demonstrates  that  the  software  architecture  and  detailed 
design,  together  with  the  computer  resources  (hardware  and  software)  that  have  been 
selected,  meet  performance  requirements  with  adequate  margins  for  this  point  in  the  life  cycle 
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g.  Updated  engineering  analyses  and  trade  studies  demonstrate  the  adequacy  of  the  software 
architecture  and  detailed  design,  together  with  the  computer  resources  (hardware  and 
software)  that  have  been  selected,  for  meeting  the  computer  resource  margin  requirements 

h.  All  the  above  analyses  take  into  account  actual  performance  of  existing  software  (e.g., 
prototypes,  earlier  builds,  NDI)  on  the  selected  hardware 

i.  Updated  engineering  models  and  simulations  have  been  used  to  demonstrate  the  adequacy  of 
the  algorithms  to  be  implemented  in  software 

5.  Integration  and  Verification 

a.  Updated  software  integration  and  qualification  test  plans  and  procedures  have  been  defined  to 
the  level  of  completeness  called  for  in  the  Software  Development  Plan,  based  on  the  selected 
software  life  cycle  model 

b.  Software  qualification  test  plans  and  procedures  are  valid,  complete,  stable,  consistent  with 
the  software  architecture,  detailed  design  and  with  higher-level  test  plans,  and  consistent  with 
the  qualification  requirements  for  test  methods  and  test  levels  for  the  software  requirements 
and  software  interface  requirements 

c.  Software  requirements  are  fully  allocated  to  the  tests  described  in  the  software  qualification 
test  plans,  where  they  will  be  verified 

d.  The  software  integration  has  been  performed  according  to  the  integration  procedures,  to  the 
level  specified  by  the  SDP  according  to  the  selected  life  cycle  model 

e.  Software  requirements  verification  status  is  documented  and  configuration  managed.  The 
status  correctly  reflects  the  results  of  the  verification  to  date,  including  the  status  of  partially 
verified  requirements,  for  all  levels  of  requirements,  from  system  through  software.  The 
verification  status  is  traced  to  the  appropriate  qualification  testing  results  (i.e.,  inspection, 
analysis,  test,  or  demonstration  reports) 

f.  The  master  software  build  plan  is  complete,  feasible,  executable,  and  consistent  with  the 
software  requirements,  software  architecture,  software  qualification  test  plans,  and  higher- 
level  schedules 

6.  Traceability 

a.  All  software  traceability  information  is  correct,  bidirectional,  and  consistent  with  the  higher- 
level  requirements  allocated  to  software,  software  requirements,  software  interface 
requirements,  software  architectural  and  detailed  design  components,  and  software 
qualification  test  plans  and  procedures 

b.  Software  traceability  information  is  defined  to  the  level  of  completeness  defined  in  the 
Software  Development  Plan,  based  on  the  selected  life  cycle  model 

7.  Risk  Management 

a.  Updated  risk  assessment  includes  the  following  software  risks  as  appropriate: 

1)  Risks  related  to  software  size  and  complexity 

2)  Risks  related  to  requirements  allocated  to  software 

3)  Risks  related  to  the  software  architecture  and  design 

4)  Risks  related  to  selection  and  use  of  NDI  (COTS,  reuse,  GOTS) 

5)  Risks  related  to  selection  and  use  of  computing  resources  (e.g.,  processors,  cache, 
memory,  buses,  and  networks) 

6)  Risks  related  to  growth  margins  for  computing  resources 

7)  Risks  related  to  software  schedules 
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8)  Risks  related  to  software  development,  integration,  and  verification  processes  and  tools 

9)  Risks  related  to  population,  update,  control,  and  validation  of  databases 

1 0)  Risks  related  to  software  and  computer  hardware  technology 

11)  Risks  related  to  software  reliability,  maintainability,  and  availability 

12)  Risks  related  to  human  systems  integration,  safety,  and  information  assurance 

13)  Updated  software  risk  management  plan  is  part  of  the  updated  SDP  and  is  integrated 
with  the  updated  system  risk  management  plan 

14)  Effective  software  risk-handling  plans  are  in  place,  and  risk- handling  activities  are  being 
performed  in  accordance  with  the  plans 

8.  Costs  and  Schedules 

a.  Software  cost  models  have  been  calibrated  with  actual  data  (both  from  the  current  project  as 
well  as  past  history)  and  used  to  update  software  cost  and  schedule  estimates 

b.  Realistic  software  cost  drivers,  such  as  complexity  and  other  parameters,  and  assumptions  are 
documented,  validated  with  documented  project  data,  and  used  in  software  cost  models  to 
develop  updated  cost  and  schedule  estimates 

c.  Updated  software  size  estimates  are  supportable,  based  on  history,  and  consistent  with  the 
software  and  interface  requirements  and  software  architecture  and  detailed  design 

d.  Software  cost  and  schedule  estimates  have  enough  margin  to  cover  the  estimation  risk 
appropriate  to  this  point  in  time 

e.  Updated  software  schedules  are  consistent  with  higher-level  schedules,  including  the  IMS 

f.  The  updated  life  cycle  cost  estimate  adequately  includes  software  support 

g.  All  of  the  software  tasks  are  included  in  the  updated  life  cycle  cost  estimates,  e.g.,  COTS 
integration  and  refresh,  screen  definition,  knowledge  base,  and  database  population 

h.  The  updated  life  cycle  cost  estimate  is  consistent  with  the  software  architecture  and  detailed 
design 

9.  Engineering  and  Management  Plans 

a.  The  updated  SDP  is  consistent  with  the  updated  IMP,  SEMP,  and  other  management  and 
engineering  plans 

b.  The  updated  SDP  addresses  the  full  software  development  life  cycle 

c.  The  updated  SDP  describes  an  integrated  set  of  processes,  methodologies,  tools,  and 
environments  that  cover  all  software  team  members,  are  suitable  for  the  domain,  and  are 
appropriate  for  program  scope  and  complexity 

d.  The  updated  SDP  describes  selected  software  development  life  cycle  models  that  are  feasible, 
appropriate  for  program  scope  and  complexity,  and  used  consistently  across  all  team 
members 

e.  Updated  software  processes,  standards,  procedures,  and  conventions  for  use  throughout  the 
life  cycle  are  documented  and  validated,  and  consistent  with  the  SDP 

f.  The  software  development  and  test  environments  integrate  with  the  systems  engineering 
environments  across  all  the  team  members 

g.  The  software  development  and  test  environments  are  established  and  have  adequate 
capability  and  capacity  to  meet  the  software  development  and  test  requirements  and  schedules 

h.  The  contractor  has  demonstrated  that  the  software  processes,  standards,  procedures,  and 
conventions  are  being  followed,  as  appropriate  to  this  point  in  the  life  cycle 
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i.  Software-related  IMP  accomplishments  for  the  CDR  have  successfully  met  their 
accomplishment  criteria 

10.  Metrics  and  Technical  Performance  Measures 

a.  Updated  definitions  for  the  selected  software  metrics  are  documented,  clear,  and  correct,  and 
include  reasonable  thresholds  for  triggering  corrective  action 

b.  Updated  software  metrics  are  sufficient  for  meeting  the  information  needs  for  program  and 
engineering  management  and  incoiporate  lessons  learned  from  the  metrics  experience  to  date 

c.  Software  metrics  are  being  collected,  analyzed,  reported,  and  used  for  management  and 
technical  decision-making,  including  risk  management,  as  appropriate  to  this  point  in  the  life 
cycle 

d.  Adequate  corrective  actions  have  been  defined  to  address  the  underlying  problems  indicated 
by  software  metrics  that  are  outside  of  documented  thresholds 

e.  TPMs  are  being  collected,  analyzed,  reported,  and  used  for  managing  the  utilization  of  all 
critical  computer  resources,  e.g.,  processors,  memory,  storage,  and  input/output  channels  and 
networks 

f.  TPMs  are  being  collected,  analyzed,  reported,  and  used  for  managing  the  software-related 
KPPs  and  driving  requirements,  including  response  time  and  timeline  requirements 

g.  Adequate  corrective  actions  have  been  defined  to  address  the  underlying  problems  indicated 
by  software  TPMs  that  are  outside  of  documented  thresholds 

h.  The  contractor  has  demonstrated  that,  for  metrics  or  TPMs  outside  of  thresholds,  corrective 
actions  have  been  initiated,  managed,  and  tracked  to  closure 

i.  The  software  problem/deficiency  report  status  indicates  that  adequate  progress  is  being  made 
in  implementing  and  verifying  solutions  to  documented  problems,  and  that  the  documented 
problems  are  being  addressed  in  accordance  with  their  severity 

R.  Data  Storage  (Security,  Access,  Distribution,  and  Delivery) 

1.  Storage  System  Capability/Flexibility/Scalability,  critical  design  maturity,  e.g.: 

a.  Analysis  identifies  needed  reliability,  maintainability,  and  availability  characteristics  of 
storage  systems  environments 

b.  Capacity,  flexibility,  and  extensibility  parameters  are  defined  that  meet  system  design  life 
requirements 

c.  Key  system  components  are  defined 

d.  Plans  for  redundancy  are  defined,  including  storage  media  hardware  and  software  capabilities 
and  types 

e.  Storage  system  management  and  performance  optimization  (including  software  management 
tools  to  provide  appropriate  partitioning/addressability)  are  defined 

f.  Analysis  defined  the  operational  environments  for  the  storage  system,  including  hardening 
levels 

2.  Storage  System  Architecture,  e.g.: 

a.  The  Storage  System  Architecture  is  defined,  including  communications  and  processing 
capacity 

b.  The  types  of  storage  system  needs  are  defined  and  fully  integrated  into  the  architecture,  e.g., 
centralized  vs.  distributed  storage;  online,  near-line,  and  offline  needs;  archive  (including 
hierarchical  storage  management,  if  appropriate),  backup,  and  restore;  and  data  replication 
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c.  Storage  hardware  components  such  as  RAID,  Storage  Area  Networks  (SAN),  Network 
Attached  Storage  (NAS),  and  Direct  Attached  Storage  (DAS)  defined  and  integrated  into  the 
architecture 

d.  Data  management  software  capabilities  are  defined  and  integrated  into  the  architecture,  e.g., 
automatic  file  migration  and  transparent  file  retrieval;  migration  between  hierarchical  levels; 
and  utilities  to  report  on  media  usage,  error  detection,  and  identification  of  media  to  be 
replaced 

3.  Security,  e.g.: 

a.  The  level  of  user  integrity  (e.g.,  access  control  lists)  is  defined 

b.  The  level  of  encryption  is  defined 

c.  The  specialized  security  capabilities,  such  as  CDS,  MLS,  and  Security  Enclaves,  are  defined 

4.  Data  Distribution  Methods,  e.g.: 

a.  A  complete  list  of  data  receivers  is  defined  that  includes  both  computer  and  human  agents 

b.  The  method(s)  of  distributing  data  to  the  various  receivers  defined,  e.g.,  method  to 
Subscribe/Publish,  Push/Pull,  and  global  or  restricted  Web-based  access 

c.  The  data  distribution  methods  are  defined  and  integrated  into  the  storage  architecture 

5.  Functionality,  e.g.: 

a.  Analysis  defined  the  physical  aspects  of  the  functionality  that  supports  the  mission 

b.  The  types  of  platforms  (server/client)  and  operating  systems  supported  are  defined 

c.  The  data  connection/transport  protocols  (e.g.,  fiber  channel,  infiniband,  SWCI)  are  defined 
and  integrated  into  the  system  architecture 

d.  Specific  reporting  (e.g.,  usage)  and  maintenance  metrics  (e.g.,  MTBF/MTTR)  are  defined 

e.  Mapping  between  metrics  and  system-level  requirements  is  complete 

50.4.5  Integrated  Technical  Risk  Management  and  Mitigation 

Evidence  of  technical  risk  management  (RM)  process  criteria  examples,  with  an  increased  level  of  fidelity 
and  maturity  of  identified  risk  items/elements,  is  provided  for  the  recommended  system  design  as  a  key 
component  of  an  integrated  program  (technical,  cost,  schedule,  and  performance)  RM  and  Mitigation 
(RM&M)  process  for  CDR,  e.g.: 

A.  Sound  risk-handling  plans,  including  risk  mitigation  processes  and  procedures,  shown  to  be  in  place 
and  executed  at  PDR  are  being  used,  maintained,  and  updated  as  required  for  the  critical  design 

1 .  The  risk- handling  plans  include  thresholds  for  taking  action,  and  appropriate  actions  have  been 
taken  when  thresholds  are  breached 

2.  The  risk  mitigation  processes  and  procedures  are  feasible,  and  alternative  courses  of  action  are 
identified 

3.  Risk-handling  plans  demonstrate  that  the  degree  of  risk  in  all  aspects  of  the  system,  segment, 
interdependency,  and  program  is  acceptable  to  proceed  to  detailed  design  of  end  items 

B.  Integrated  system-level  risk  reduction  approaches  and  processes  developed  by  PDR  are  verified  and 
baselined  for  the  critical  design  and  can  encompass  such  items  as: 

1.  The  significant  program- level  risks  (technical/performance,  cost,  and  schedule) 

2.  Risk  management  database/tools  for  risk  metrics  collection,  analysis,  tracking,  and  reporting 

3.  Risk  mitigation/reduction  strategies,  bum-down  plans  that  are  linked  to  dependencies  to  the 
Program  IMP/IMS  and  WBS 

4.  Continuous  risk  monitoring/review,  identification,  assessment,  and  ranking 
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5.  Technology  and  manufacturing  readiness  level  (TRL  and  MRL)  assessments  and  metrics 

6.  Requirements  risk  assessment  metrics 

7.  Critical  risk  management  of  software  issues,  e.g.,  complexity,  size,  processing  speed, 
throughput,  schedules,  COTS  availability,  legacy  reuse  suitability,  and  software  development 
processes  and  tools 

8.  A  comprehensive  risk  assessment  for  the  follow-on  phases 

9.  TRL  and  MRL  assessment  metrics 
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Appendix  F  -  Test  Readiness  Review  (TRR) 


60.  Test  Readiness  Review  (TRR) 

The  TRR  is  a  multifunctional  and  multidisciplinary  review  and  process  assessment  to  verify  the 
contractor’s  readiness  to  begin  a  formal  verification  testing  event  for  an  EL 

60.1  General 

A  TRR  shall  be  conducted  at  the  beginning  of  each  formal  verification  testing  event.  The  TRR  shall  occur 
after  the  test  procedures  for  verifying  the  El  requirements  are  prepared  and  a  successful  dry  run  of  the  test 
procedures  has  occurred,  and  before  any  formal  “run  for  the  record”  begins.  As  an  alternate  to  or  tailoring 
of  the  “dry  run”  followed  by  a  “run  for  the  record,”  it  is  acceptable  to  consider  the  use  of  a 
multidisciplinary  test  procedure  validation  team  to  validate  (redline)  a  test  procedure’s  first  run  while 
simultaneously  performing  the  “run  for  the  record.”  The  TRR  shall  be  held  for  an  individual  El  or  a 
collection  of  related  Els,  as  defined  or  tailored  by  the  contractor  and  approved  by  the  contracting  agency. 

60.2  Objectives 

The  objective  of  the  TRR  is  to  determine  whether  the  contractor  is  ready  to  begin  a  formal  verification 
testing  event  for  one  or  more  Els.  This  includes  an  assessment  of  whether: 

a.  The  El  is  sufficiently  mature  to  begin  testing 

b.  The  test  procedures,  together  with  the  test  data,  are  sufficiently  robust  to  verify  the 
requirements 

c.  The  test  procedures  have  been  successfully  dry  run 

d.  Disciplined  test  processes  are  in  place 

e.  Test  personnel  are  available,  and  roles  and  responsibilities  are  clearly  defined 

60.3  Items  To  Be  Reviewed 

The  contractor  shall  present  the  following  items  for  review  by  the  contracting  agency  for  the  EI(s)  under 
review: 

A.  Requirements 

1 .  El  requirements  are  planned  to  be  verified  at  this  formal  verification  event 

2.  Any  changes  to  these  El  requirements  that  have  been  approved  since  CDR 

3.  Required  verification  methods  (Inspection  (I),  Analysis  (A),  Demonstration  (D),  Test  (T))  and 
verification  levels  for  each  El  requirement  are  planned  to  be  verified  at  this  formal  verification 
event 

B.  Design 

1 .  Any  changes  to  the  El  design  that  have  occurred  since  CDR  and  that  affect  the  El  verification 
testing 

C.  Test  Plans 

1 .  Any  changes  to  the  El  test  plan  that  have  occurred  since  CDR 

D.  Test  Procedures 

1.  Test  cases  and  test  procedures  for  the  formal  verification  event  to  be  held,  including  but  not 
limited  to: 

a.  Test  setup  procedures 

b.  Test  execution  procedures 

c.  Data  capture  procedures 
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d.  Data  analysis  procedures 

2.  Description  of  the  test  data  to  be  used  with  the  test  procedures 

3.  Results  of  dry  runs,  including  anomalies  encountered  and  any  anticipated  problems  with 
requirements  verification.  Evidence  of  test  procedure  adequacy  reviews  or  peer  reviews  of  the 
test  procedure  is  acceptable  if  tailored  by  contract  as  an  alternative  to  a  “dry  run”  if  approach 
consists  of  test  procedure  validation  concurrent  with  the  run  for  the  record  approach 

E.  Traceability 

1.  Traceability  between  the  test  cases  and  test  procedures  to  be  executed  and  the  requirements  to 
be  verified  by  each 

F.  Description  of  the  El 

1 .  Description  of  the  El  hardware  and  software  under  test 

2.  Configuration  of  the  El  hardware  and  software  under  test,  including  the  specific  version  or 
release  of  the  software,  e.g.,  confirmation  of  the  configuration  of  the  hardware  and  software 
under  test  by  the  CM  organization 

3.  User  documentation  for  the  El,  if  applicable  (e.g.,  user  manuals,  handbooks) 

G.  El  Problems  and  Deficiencies 

1.  All  known  El  hardware  and  software  problems  or  deficiencies  as  of  the  start  of  the  verification 
event,  with  their  severity  levels 

2.  Expected  impact  of  the  known  problems  or  deficiencies  on  the  testing 

H.  Test  Environment 

1.  Description  of  the  test  environment,  including  hardware,  software,  automated  test  equipment, 
test  tools,  simulators,  emulators,  drivers,  etc. 

2.  Confirmation  of  the  configuration  of  the  test  environment  by  the  CM  organization 

3.  Status  of  validation  performed  for  test  environment  components  (including  hardware,  software, 
automated  test  equipment,  test  tools,  simulators,  emulators,  drivers,  etc.)  to  ensure  they 
correctly  perform  the  functions  necessary  to  support  the  verification  testing  event 

4.  All  known  test  environment  hardware  and  software  problems  or  deficiencies  and  their  expected 
impact  on  the  testing 

I.  Processes,  Roles,  Responsibilities,  and  Authorities 

1 .  Test  personnel  and  their  roles,  responsibilities,  and  authorities: 

a.  Including  CM  and  QA  personnel  as  well  as  test  team  personnel 

b.  Including  both  Government  and  contractor 

2.  Test  processes  to  be  followed,  including: 

a.  A  nominal  process 

b.  A  retest  process  when  test  anomalies  are  encountered  and  corrections  must  be  performed 

c.  An  anomaly  adjudication  process  to  determine  whether  and  how  testing  can  be  continued 
after  an  anomaly  has  occurred  during  execution 

d.  A  record  maintenance  process  of  the  El  requirements’  verification  status  (fully  verified, 
partially  verified,  not  verified),  following  completion  of  test  execution  and  data  analysis 

J.  Test  schedules 

1 .  Hour-by-hour  schedules  for  the  verification  test  event 

2.  Justification  for  schedules  based  on  dry-run  test  times 
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K.  Test  Limitations 

1 .  Any  test  limitations  or  other  conditions  that  might  impact  the  testing 

60.4  TRR  “Acceptance  Criteria” 

A.  Requirements 

1.  The  requirements  planned  to  be  verified  at  this  test  verification  event  include  all  approved 
changes 

B.  Design 

1 .  Any  design  changes  made  since  CDR  shall  not  adversely  affect  this  test  verification  event 

C.  Test  Plans 

1 .  All  changes  to  the  test  plan  have  been  approved 

2.  The  test  plan,  with  all  changes  incorporated,  remains  sufficiently  robust  to  ensure  the  full 
verification  of  all  requirements 

3.  The  test  plan  is  consistent  with  the  required  verification  methods  and  levels 

D.  Test  Procedures 

1.  The  test  procedures  for  each  test  case,  together  with  the  test  data,  are  correct,  complete,  and 
sufficiently  robust  to  verify  all  of  the  requirements  allocated  to  the  test  case 

2.  The  test  procedures  are  consistent  with  the  required  verification  methods  and  levels 

3.  The  test  procedures  are  sufficiently  detailed  to  be  repeatable 

4.  The  test  procedures  are  in  compliance  with  the  approved  test  plan 

5.  All  redlines  from  dry  runs  have  been  incoiporated  into  the  test  procedures 

E.  Traceability 

1.  Bidirectional  traceability  is  provided  between  the  requirements  under  test  and  the  test  cases  and 
test  procedures  in  which  the  requirements  will  be  verified 

2.  The  traceability  is  correct,  complete,  and  consistent 

3.  Identification  of  the  test  procedures’  steps  is  provided  where  the  verification  of  each 
requirement  is  completed 

F.  Description  of  the  El 

1 .  The  El  under  test  is  clearly  defined 

2.  The  El  is  under  configuration  control  by  the  CM  organization,  and  the  configuration  of  each  El 
hardware  and  software  component  is  documented 

G.  El  Problems  and  Deficiencies 

1 .  The  El,  including  hardware  and  software,  is  sufficiently  mature  to  begin  the  verification  testing 
event 

2.  No  severity  1  or  2  problems  or  deficiencies  are  open  for  the  portion  of  the  El  undergoing  test 

H.  Test  Environment 

1.  The  test  environment,  including  hardware  and  software,  is  sufficiently  robust  to  adequately 
verify  the  requirements  under  test 

2.  Sufficient  validation  has  occurred  prior  to  the  verification  test  event  to  ensure  that  the  test 
environment,  including  hardware  and  software,  will  correctly  perform  the  functions  necessary 
to  support  the  verification  testing  event 
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3.  The  test  environment  is  under  configuration  control  by  the  CM  organization,  and  the 
configuration  of  each  component  of  the  test  environment  is  documented 

I.  Processes,  Roles,  and  Responsibilities 

1 .  Processes  to  be  followed  during  test  execution  are  defined  and  documented  and  will  result  in  a 
controlled  and  disciplined  test  execution 

2.  Personnel  roles  and  responsibilities  are  well  defined,  both  for  the  contractor  and  Government 

3.  The  presence  and  role  of  QA  personnel  is  sufficient  to  ensure: 

a.  The  test  process  is  followed 

b.  Test  execution  rigorously  follows  the  test  procedures  with  any  deviations  documented  as 
redlines 

c.  All  problems  or  deficiencies  encountered  during  testing  are  documented  on  the  appropriate 
forms 

d.  The  test  log  faithfully  documents  the  execution  of  the  test,  including  test  start,  test  end, 
interruptions,  and  anomalies 

J.  Test  Schedules 

1.  Test  schedules  are  feasible 

K.  Test  Limitations 

1 .  Any  test  limitations  will  not  affect  the  ability  to  verify  the  planned  requirements 
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70.  Functional  Configuration  Audit  (FCA) 

For  Space  Systems  specific  guidance,  see  SMC  STD  SMCS-S-002  Configuration  Management  dated  13 

June  2008. 

70.1  General 

A.  The  objective  of  the  Functional  Configuration  Audit  (FCA)  is  to  verify  that  the  configuration  item’s 
actual  performance  complies  with  its  hardware  Development  or  Software  Requirements  and 
Interface  Requirements  Specifications.  Test  data  is  be  reviewed  to  verify  that  the  hardware  or 
computer  software  performs  as  required  by  its  functional  and  allocated  configuration  identification. 
For  configuration  items  developed  at  government  expense,  an  FCA  shall  be  a  prerequisite  to 
acceptance  of  the  configuration  item.  For  software,  a  technical  understanding  shall  be  reached  on 
the  validity  and  the  degree  of  completeness  of  the  Software  Test  Reports,  and  as  appropriate, 
Computer  System  Operator’s  Manual  (CSOM),  Software  User’s  Manual  (SUM),  and  the  Computer 
System  Diagnostic  Manual  (CSDM). 

B.  The  FCA  for  a  complex  configuration  item  shall  be  conducted  on  a  progressive  basis,  when  so 
specified  by  the  contracting  agency,  throughout  the  configuration  item’s  development  and 
culminates  at  the  completion  of  the  qualification  testing  of  the  configuration  item  with  a  review  of 
all  discrepancies  at  the  final  FCA.  The  FCA  is  to  be  conducted  on  that  configuration  of  the 
configuration  item  that  is  representative  (prototype  or  preproduction)  of  the  configuration  to  be 
released  for  production  of  the  operational  inventory  quantities.  When  a  prototype  or  preproduction 
article  is  not  produced,  the  FCA  shall  be  conducted  on  a  first  production  article.  For  cases  where 
configuration  item  qualification  can  be  determined  only  through  integrated  system  testing,  FCAs 
for  such  configuration  items  will  not  be  considered  complete  until  completion  of  such  integrated 
testing. 

C.  Recommendations  of  configuration  item  acceptance  or  nonacceptance  to  the  local  contract 
management  agency  are  based  upon  and  governed  by  procedures  and  requirements  outlined  in 
subsequent  paragraphs. 

D.  Continuing  with  the  results  of  the  Critical  Design  Review  (CDR),  review  engineering  data  as 
defined  in  Paragraph  3.27  as  to  the  suitability  for  intended  use.  The  review  shall  consider  the 
checklist  items  discussed  in  Paragraph  100.6,  as  properly  tailored. 

70.2  Contract  Requirements 

A.  The  schedules  for  the  FCA  shall  be  recorded  on  the  configuration  item  development  record  by  the 
contractor.  A  configuration  item  cannot  be  audited  without  the  contracting  agency  authentication  of 
the  functional  and  allocated  baseline.  In  addition,  the  contractor  shall  submit  the  final  draft  Product 
Specification  for  the  configuration  item  to  be  audited  to  the  contracting  agency  for  review  prior  to 
FCA. 

70.3  Contractor  Responsibility 

A.  Prior  to  the  FCA  date  (for  configuration  items  to  be  audited),  the  contractor  shall  provide  the 
following  information  to  the  contracting  agency  (this  information  is  to  be  provided  in  addition  to 
the  general  requirements  of  Sections  4  and  5): 

1 .  Contractor  representation  (the  test  manager  shall  be  in  attendance) 

2.  Identification  of  items  to  be  audited: 

a.  Nomenclature 

b.  Specification  identification  number 

c.  Configuration  item  number 
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d.  Current  listing  of  all  deviations/waivers  against  the  configuration  item,  either  requested  of, 
or  approved  by  the  contracting  agency 

e.  Status  of  Test  Program  to  test  configured  items  with  automatic  test  equipment  (when 
applicable) 

70.4  Procedures  and  Requirements 

The  contractor’s  test  procedures  and  results  shall  be  reviewed  for  compliance  with  specification 
requirements  as  mapped  down  to  the  lowest  level  of  El  in  a  requirements  test  verification  matrix 
(RTVM). 

A.  The  following  testing  information  shall  be  available  for  the  FCA  team: 

1.  Test  plans,  specifications,  descriptions,  procedures,  and  reports  for  the  configuration  item 

2.  A  complete  list  of  successfully  accomplished  functional  tests  during  which  pre-acceptance  data 
was  recorded 

3.  A  complete  list  of  successful  functional  tests  if  detailed  test  data  is  not  recorded 

4.  A  complete  list  of  functional  tests  required  by  the  specification  but  not  yet  performed.  (To  be 
performed  as  a  system  or  subsystem  test) 

5.  Preproduction  and  production  test  results 

B.  Testing  accomplished  with  the  approved  test  procedures  and  validated  data  (witnessed)  should  be 
sufficient  to  ensure  configuration  item  performance  as  set  forth  in  the  specification  Section  3  and 
meet  the  quality  assurance  provisions  and  qualification  requirements  contained  in  Section  4 

C.  For  those  performance  parameters  that  cannot  completely  be  verified  during  testing,  adequate 
analysis  or  simulation  shall  have  been  accomplished.  The  results  of  the  analysis  or  simulations  will 
be  sufficient  to  ensure  configuration  item  performance  as  outlined  in  the  specification. 

D.  Test  reports,  procedures,  and  data  used  by  the  FCA  team  are  to  be  made  a  matter  of  record  in  the 
FCA  minutes 

E.  A  list  of  the  contractor’s  internal  documentation  (drawings)  of  the  configuration  item  is  to  be 
reviewed  to  ensure  that  the  contractor  has  documented  the  physical  configuration  of  the 
configuration  item  for  which  the  test  data  is  verified 

F.  Drawings  of  HWCI  parts  that  are  to  be  provisioned  should  be  selectively  sampled  to  assure  that  test 
data  essential  to  manufacturing  are  included  on,  or  furnished  with,  the  drawings 

G.  Configuration  Items  (CIs)  that  fail  to  pass  quality  assurance  test  provisions  are  to  be  analyzed  as  to 
the  cause  of  failure  to  pass.  Appropriate  corrections  shall  be  made  to  both  the  Cl  and  associated 
engineering  data  before  a  Cl  is  subjected  to  requalification. 

H.  A  checklist  shall  be  developed  that  identifies  documentation  and  hardware  and  computer  software 
to  be  available  and  tasks  to  be  accomplished  at  the  FCA  for  the  configuration  item.  See  Pre-FCA 
check  sheet. 

I.  Retests  or  additional  tests  shall  be  performed  to  assure  compliance  with  paragraph  70.4.3 

J.  Partial  completion  of  the  FCA  for  those  configuration  items  whose  qualification  is  contingent  upon 
completion  of  integrated  systems  testing  is  acknowledged 

K.  For  SWCIs  the  following  additional  requirements  shall  apply: 

1.  The  contractor  is  to  provide  the  FCA  team  with  a  briefing  for  each  SWCI  being  audited  and 
delineate  the  test  results  and  findings  for  each  SWCI.  As  a  minimum,  the  discussion  is  to 
include  SWCI  requirements  that  were  not  met,  including  a  proposed  solution  to  each  item,  an 
account  of  the  ECPs  incoiporated  and  tested  as  well  as  proposed,  and  a  general  presentation  of 
the  entire  SWCI  test  effort,  delineating  problem  areas  as  well  as  accomplishments 
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2.  An  audit  of  the  formal  test  plans,  descriptions,  and  procedures  are  to  be  made  and  compared 
against  the  official  test  data.  The  results  are  to  be  checked  for  completeness  and  accuracy. 
Deficiencies  are  to  be  documented  and  made  a  part  of  the  FCA  minutes.  Completion  dates  for 
all  discrepancies  are  to  be  clearly  established  and  documented 

3.  An  audit  of  the  Software  Test  Reports  are  to  be  performed  to  validate  that  the  reports  are 
accurate  and  completely  describe  the  SWCI  tests 

4.  All  ECPs  that  have  been  approved  are  to  be  reviewed  to  ensure  that  they  have  been  technically 
incorporated  and  verified 

5.  All  updates  to  previously  delivered  documents  are  to  be  reviewed  to  ensure  accuracy  and 
consistency  throughout  the  documentation  set 

6.  Preliminary  and  Critical  Design  Review  minutes  are  to  be  examined  to  ensure  that  all  findings 
have  been  incorporated  and  completed 

7.  The  interface  requirements  and  the  testing  of  these  requirements  are  to  be  reviewed  for  SWCIs 

8.  Review  database  characteristics,  storage  allocation  data  and  timing,  and  sequencing 
characteristics  for  compliance  with  specified  requirements 

70.5  Post-Audit  Actions 

A.  After  completion  of  a  specific  FCA,  the  contractor  shall  publish  and  distribute  copies  of  FCA 
minutes.  The  contracting  agency  officially  acknowledges  completion  of  the  FCA  with  the 
indication  that  successful  performance  of  an  FCA  satisfies  the  requirements  for  the  conduct  of  the 
PCA. 

B.  The  accomplishment  of  the  FCA  shall  be  recorded  on  the  configuration  item  Development  Record 
by  the  contractor. 
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Appendix  H  -  Physical  Configuration  Audit  (PCA) 


80.  Physical  Configuration  Audit  (PCA) 

For  Space  Systems-specific  guidance,  see  SMC  STD  SMCS-S-002  Configuration  Management  dated  13 
June  2008. 

80.1  General 

A.  The  Physical  Configuration  Audit  (PCA)  shall  be  the  formal  examination  of  the  as-built  version  of 

a  configuration  item  against  its  design  documentation  in  order  to  establish  the  product  baseline. 
After  successful  completion  of  the  audit,  all  subsequent  changes  are  processed  by  engineering 
change  action.  The  PCA  also  determines  that  the  acceptance  testing  requirements  prescribed  by  the 
documentation  is  adequate  for  acceptance  of  production  units  of  a  configuration  item  by  quality 
assurance  activities.  The  PCA  includes  a  detailed  audit  of  engineering  drawings,  specifications, 

technical  data,  and  tests  utilized  in  the  production  of  HWCIs  and  a  detailed  audit  of  design 

documentation,  listings,  and  manuals  for  SWCIs.  The  review  shall  include  an  audit  of  the  released 
engineering  documentation  and  quality  control  records  to  make  sure  the  as-built  or  as-coded 
configuration  is  reflected  in  this  documentation.  For  software,  the  Software  Product  Specification 
and  Software  Version  Description  shall  be  a  part  of  the  PCA  review. 

B.  The  PCA  shall  be  conducted  on  the  first  article  of  configuration  items,  and  those  that  are  a 
reprocurement  of  a  configuration  item  already  in  the  inventory  will  be  identified  and  selected 
jointly  by  the  contracting  agency  and  the  contractor.  A  PCA  shall  be  conducted  on  the  first 
configuration  item  to  be  delivered  by  a  new  contractor  even  though  PCA  was  previously 
accomplished  on  the  first  article  delivered  by  a  different  contractor. 

C.  Formal  approval  by  the  contracting  agency  of  the  configuration  item  Product  specification  and  the 
satisfactory  completion  of  a  PC  A  result  in  establishment  of  the  product  baseline. 

D.  Recommendations  of  configuration  item  acceptance  or  nonacceptance  to  the  responsible  contract 
administration  office  (CAO)  are  based  upon  and  governed  by  procedures  and  requirements 
outlined  in  subsequent  paragraphs. 

E.  A  final  review  will  be  made  of  all  operation  and  support  documents  (i.e.,  Computer  System 
Operator’s  Manual  (CSOM),  Software  User’s  Manual  (SUM),  Computer  System  Diagnostic 
Manual  (CSDM),  Software  Programmer’s  Manual  (SPM),  Firmware  Support  Manual  (FSM))  to 
check  format,  completeness,  and  conformance  with  applicable  data  item  descriptions. 

F.  Continuing  with  the  results  of  the  Functional  Configuration  Audit  (FCA),  review  engineering  data 
as  defined  in  Paragraph  3.15  as  to  the  suitability  for  intended  use.  The  review  should  consider  the 
checklist  items  discussed  in  Paragraph  100.6,  as  properly  tailored. 

80.2  Contract  Requirements 

The  schedules  for  the  PCA  shall  be  recorded  on  the  configuration  item  Development  Record  by  the 
contractor.  A  set  of  current  listings  shall  be  provided  for  each  SWC1  being  audited.  The  contractor  shall 
submit  the  final  draft  of  the  product  specification  for  the  configuration  item  to  be  audited  to  the 
contracting  agency  for  review  prior  to  PCA. 

80.3  Contractor  Responsibility 

A.  The  contractor  shall  provide  the  following  information  to  the  contracting  agency  (this  information 
shall  be  provided  in  accordance  with  the  general  instructions  of  Sections  4  and  5  and  the 
contractual  requirements): 

1 .  Contractor  representation  (the  test  manager  should  be  in  attendance) 
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2.  Identification  of  items  to  be  accepted  by: 

a.  Nomenclature 

b.  Specification  identification  number 

c.  Configuration  item  identifiers 

d.  Serial  numbers 

e.  Drawing  and  part  numbers 

f.  Identification  numbers 

g.  Code  identification  numbers 

h.  Software  inventory  numbering  system 

3.  A  list  delineating  all  deviations/waivers  against  the  configuration  item  either  requested  or 
contracting  agency  approved 

B.  The  PCA  cannot  be  performed  unless  data  pertinent  to  the  configuration  item  being  audited  is 
provided  to  the  PCA  team  at  time  of  the  audit.  The  contractor  is  to  compile  and  make  this 
information  available  for  ready  reference.  Required  information  is  to  include: 

1 .  Configuration  item  product  specification 

2.  A  list  delineating  both  approved  and  outstanding  changes  against  the  configuration  item 

3.  Complete  shortage  list 

4.  Acceptance  test  procedures  and  associated  test  data 

5.  Engineering  drawing  index,  including  revision  letters 

6.  Operating,  maintenance,  and  illustrated  parts  breakdown  manuals 

7.  Proposed  DD  Form  250,  “Material  Inspection  and  Receiving  Report” 

8.  Approved  nomenclature  and  nameplates 

9.  Software  Programmer’s  Manuals  (SPMs),  Software  User’s  Manuals  (SUMs),  Computer  System 
Operator’s  Manual  (CSOM),  Computer  System  Diagnostic  Manual  (CSDM),  and  Firmware 
Support  Manual  (FSM) 

10.  Software  version  description  document 

11.  FCA  minutes  for  each  configuration  item 

12.  Findings  and  status  of  quality  assurance  programs 

C.  The  contractor  is  to  assemble  and  make  available  to  the  PCA  team  at  time  of  audit  all  data 
describing  the  item  configuration.  Item  configuration  data  is  to  include: 

1.  Current  approved  issue  of  hardware  development  specification,  Software  Requirements 
Specification,  and  Interface  Requirements  Specification(s)  to  include  approved  specification 
change  notices  and  approved  deviations  and  waivers 

2.  Identification  of  all  changes  actually  made  during  test 

3.  Identification  of  all  required  changes  not  completed 

4.  All  approved  drawings  and  documents  by  the  top  drawing  number  as  identified  in  the 
configuration  item  product  specification.  All  drawings  are  to  be  of  the  category  and  form 
specified  in  the  contract 

5.  Manufacturing  instruction  sheets  for  HWCIs  are  identified  by  the  contracting  agency 

D.  The  contractor  is  to  identify  any  difference  between  the  physical  configurations  of  the  selected 
production  unit  and  the  Development  Unit(s)  used  for  the  FCA  and  certify  or  demonstrate  to  the 
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government  that  these  differences  do  not  degrade  the  functional  characteristics  of  the  selected 

units. 

80.4  PCA  Procedures  and  Requirements 

A.  Drawing  and  Manufacturing  Instruction  Sheet  Review  Instructions: 

1 .  A  representative  number  of  drawings  and  associated  manufacturing  instruction  sheets  for  each 
item  of  hardware,  identified  by  the  contracting  agency  co-chairperson,  shall  be  reviewed  to 
determine  their  accuracy  and  ensure  that  they  include  the  authorized  changes  reflected  in  the 
engineering  drawings  and  the  hardware.  Unless  otherwise  directed  by  the  contracting  agency 
co-chairperson,  inspection  of  drawings  and  associated  manufacturing  instruction  sheets  may  be 
accomplished  on  a  valid  sampling  basis.  The  purpose  of  this  review  is  to  ensure  that  the 
manufacturing  instruction  sheets  accurately  reflect  all  design  details  contained  in  the  drawings. 
Since  the  hardware  is  built  in  accordance  with  the  manufacturing  instruction  sheets,  any 
discrepancies  between  the  instruction  sheets  and  the  design  details  and  changes  in  the  drawings 
will  also  be  reflected  in  the  hardware 

2.  The  following  minimum  information  shall  be  recorded  for  each  drawing  reviewed: 

a.  Drawing  number/title  (include  revision  letter) 

b.  Date  of  drawing  approval 

c.  List  of  manufacturing  instruction  sheets  (numbers  with  change  letter/titles  and  date  of 
approval)  associated  with  this  drawing 

d.  Discrepancies/comments 

e.  Select  a  sample  of  part  numbers  reflected  on  the  drawing.  Check  to  ensure  compatibility 
with  the  Program  Parts  Selection  List,  and  examine  the  HWCI  to  ensure  that  the  proper 
parts  are  actually  installed 

3.  As  a  minimum,  the  following  inspections  shall  be  accomplished  for  each  drawing  and 
associated  manufacturing  instruction  sheets: 

a.  Drawing  number  identified  on  manufacturing  instruction  sheet  should  match  latest  released 
drawing 

b.  List  of  materials  on  manufacturing  instruction  sheets  should  match  materials  identified  on 
the  drawing 

c.  All  special  instructions  called  out  on  the  drawing  should  be  on  the  manufacturing 
instruction  sheets 

d.  All  dimensions,  tolerances,  finishes,  etc.,  called  out  on  the  drawing  should  be  identified  on 
the  manufacturing  instruction  sheets 

e.  All  special  processes  called  out  on  the  drawing  should  be  identified  on  the  manufacturing 
instruction  sheets 

f.  Nomenclature  descriptions,  part  numbers,  and  serial  number  markings  called  out  on  the 
drawing  should  be  identified  on  the  manufacturing  instruction  sheets 

g.  Review  drawings  and  associated  manufacturing  instruction  sheets  to  ascertain  that  all 
approved  changes  have  been  incorporated  into  the  configuration  item 

h.  Check  release  record  to  ensure  that  all  drawings  reviewed  are  identified 

i.  Record  the  number  of  any  drawings  containing  more  than  five  outstanding  changes 
attached  to  the  drawing 

j.  Check  the  drawings  of  a  major  assembly  and/or  black  box  of  the  hardware  configuration 
item  for  continuity  from  top  drawing  down  to  piece-part  drawing 
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B.  Review  of  all  records  of  baseline  configuration  for  the  HWCI  by  direct  comparison  with 
contractor’s  engineering  release  system  and  change  control  procedures  to  establish  that  the 
configuration  being  produced  does  accurately  reflect  released  engineering  data.  This  includes 
interim  releases  of  spares  provisioned  prior  to  PCA  to  ensure  the  delivery  of  currently  configured 
spares. 

C.  Audit  of  contractor’s  engineering  release  and  change  control  system  to  ascertain  that  they  are 
adequate  to  properly  control  the  processing  and  formal  release  of  engineering  changes.  The 
minimum  needs  and  capabilities  set  forth  below  are  required  of  the  contractor’s  engineering  release 
records  system.  The  contractor’s  formats,  systems,  and  procedures  are  to  be  used.  Information  in 
addition  to  the  basic  requirements  shall  be  considered  part  of  the  contractor’s  internal  system.  (*) 

(*)  Contract  Administration  Office  (CAO)  Quality  Assurance  Representative  (QAR)  records  can  be 
reviewed  for  the  purpose  of  determining  the  contractor’s  present  and  most  recent  past  performance. 

D.  As  a  minimum,  the  following  information  shall  be  contained  on  one  release  record  supplied  by  the 
contractor,  subcontractor,  or  vendor  for  each  drawing  number,  if  applicable: 

1 .  Serial  numbers,  top  drawing  number,  specification  number 

2.  Drawing  number,  title,  code  number,  number  of  sheets,  date  of  release,  change  letter,  date  of 
change  letter  release,  engineering  change  order  (ECO)  number 

E.  The  contractor’s  release  function  and  documentation  shall  be  capable  of  determining: 

1.  The  composition  of  any  part  at  any  level  in  terms  of  subordinate  part  numbers  (disregard 
standard  parts) 

2.  The  next  higher  assembly  using  the  part  number,  except  for  assembly  into  standard  parts 

3.  The  composition  of  the  configuration  item  or  part  number  with  respect  to  other  configuration 
items  or  part  numbers 

4.  The  configuration  item  and  associated  serial  number  on  which  subordinate  parts  are  used.  (This 
does  not  apply  to  contractors  below  prime  level  who  are  not  producing  configuration  items.) 

5.  The  accountability  of  changes  that  have  been  partially  or  completely  released  against  the 
configuration  item 

6.  The  configuration  item  and  serial  number  effectivity  of  any  change 

7.  The  standard  specification  number  or  standard  part  numbers  used  within  any  nonstandard  part 
number 

8.  The  contractor  specification  document  and  specification  control  numbers  associated  with  any 
subcontractor,  vendor,  or  supplier  part  number 

F.  The  engineering  release  system  and  associated  documentation  shall  be  capable  of: 

1.  Identifying  changes  and  retaining  records  of  superseded  configurations  formally  accepted  by 
the  contracting  agency 

2.  Identifying  all  engineering  changes  released  for  production  incorporation.  These  changes  are  to 
be  completely  released  and  incorporated  prior  to  formal  acceptance  of  the  configuration  item 

3.  Determining  the  configuration  released  for  each  configuration  item  at  the  time  of  formal 
acceptance 

G.  Engineering  data  shall  be  released  or  processed  through  a  central  authority  to  ensure  coordinated 
action  and  preclude  the  unilateral  release  of  data 

H.  Engineering  change  control  numbers  shall  be  unique 

I.  The  difference  between  the  configuration  of  the  configuration  item  qualified  and  the  configuration 
item  being  audited  shall  be  a  matter  of  record  in  the  minutes  of  the  PC  A 


Page  123  of  168 


Appendix  H 

J.  For  F1WCI  acceptance  tests,  data  and  procedures  shall  comply  with  product  specification.  The  PCA 
team  shall  determine  any  acceptance  tests  to  be  redone,  and  reserves  the  prerogative  to  have 
representatives  of  the  contracting  agency  witness  all  or  any  portion  of  the  required  audits, 
inspections,  or  tests 

K.  FlWCIs  that  fail  to  pass  acceptance  test  requirements  shall  be  repaired  if  necessary  and  be  retested 
by  the  contractor  in  the  manner  specified  by  the  PCA  team  leader  in  accordance  with  the  product 
specification 

L.  The  contractor  shall  present  data  confirming  the  inspection  and  test  of  subcontractor  equipment 
end  items  at  point  of  manufacture.  Such  data  shall  have  been  witnessed  by  a  government 
representative 

M.  The  PCA  team  reviews  the  prepared  backup  data  (all  initial  documentation  that  accompanies  the 
configuration  item)  for  correct  types  and  quantities  to  ensure  adequate  coverage  at  the  time  of 
shipment  to  the  user 

N.  Configuration  items  that  have  demonstrated  compliance  with  the  product  specification  are 
approved  for  acceptance  as  follows: 

1.  The  PCA  team  shall  certify  by  signature  that  the  configuration  item  has  been  built  in 
accordance  with  the  drawings  and  specifications 

O.  As  a  minimum,  the  following  actions  shall  be  performed  by  the  PCA  team  on  each  SWC1  being 
audited: 

1.  Review  all  documents  that  will  make  up  the  software  product  specification  for  format  and 
completeness 

2.  Review  FCA  minutes  for  recorded  discrepancies  and  actions  taken 

3.  Review  the  design  descriptions  for  proper  entries,  symbols,  labels,  tags,  references,  and  data 
descriptions 

4.  Compare  top-level  software  units  design  descriptions  with  lower-level  software  unit 
descriptions  for  consistency 

5.  Compare  all  lower-level  design  descriptions  with  all  software  listings  for  accuracy  and 
completeness 

6.  Check  Software  User’s  Manual(s),  Software  Programmer’s  Manual,  Computer  System 
Operator’s  Manual,  Firmware  Support  Manual,  and  Computer  System  Diagnostic  Manual  for 
format  completeness  and  conformance  with  applicable  data  item  descriptions.  (Formal 
verification  and  acceptance  of  these  manuals  should  be  withheld  until  system  testing  to  ensure 
that  the  procedural  contents  are  correct.) 

7.  Examine  actual  SWCI  delivery  media  (card  decks,  tapes,  disks,  etc.)  to  ensure  conformance 
with  Section  5  of  the  Software  Requirements  Specification 

8.  Review  the  annotated  listings  for  compliance  with  approved  coding  standards 

80.5  Post  Audit  Actions 

A.  Contracting  agency  acceptance  or  rejection  of  the  configuration  item  and  the  configuration  item 
product  specification  presented  for  PCA  shall  be  furnished  to  the  contractor  in  writing  by  the 
responsible  contract  management  agency  or  other  designated  agency  after  completion  of  PCA, 
including  appropriate  corrective  actions  for  resolution  of  deficiencies 

B.  After  completion  of  the  PCA,  the  contractor  shall  publish  and  distribute  copies  of  PCA  minutes. 
The  contracting  agency  officially  acknowledges  completion  of  the  PCA  as  indicated  in  paragraph 
4.2.4 
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C.  The  accomplishment  of  the  PCA  shall  be  recorded  on  the  configuration  item  Development  Record 
by  the  contractor.  Only  the  successful  verification/validation  of  the  FBL  and  close-out  of 
corrective  action  can  result  in  authorization  for  production  go-ahead 
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Appendix  I  -  System  Verification  Review  (SVR) 


90.  System  Verification  Review  (SVR) 

System  Verification  Review  (SVR)  was  previously  identified  as  Formal  Qualification  Review  (FQR). 

90.1  General 

The  objective  of  the  SVR  shall  be  to  verify  that  the  actual  performance  of  the  configuration  items  of  the 
system  as  determined  through  test  comply  with  the  hardware  Development  Specification,  Software 
Requirements  and  Interface  Requirements  Specifications,  and  to  identify  the  test  report(s)/data  that 
documents  results  of  qualification  tests  of  the  configuration  items.  The  point  of  government  certification 
will  be  determined  by  the  contracting  agency  and  will  depend  upon  the  nature  of  the  program,  risk  aspects 
of  the  particular  hardware  and  software,  and  contractor  progress  in  successfully  verifying  the 
requirements  of  the  configuration  items.  When  feasible,  the  SVR  shall  be  combined  with  the  FCA  at  the 
end  of  configuration  item/subsystem  testing,  prior  to  PC  A.  If  sufficient  test  results  are  not  available  at  the 
FCA  to  ensure  that  the  configuration  items  will  perform  in  their  system  environment,  the  SVR  shall  be 
conducted  (post-PCA)  during  System  testing  whenever  the  necessary  tests  have  been  successfully 
completed  to  enable  certification  of  configuration  items.  For  non-combined  FCA/SVR,  traceability, 
correlation,  and  completeness  of  the  SVR  shall  be  maintained  with  the  FCA  and  duplication  of  effort 
avoided. 

90.2  Requirements 

A.  In  cases  where  the  SVR  and  the  FCA  can  be  accomplished  in  a  single  combined  Audit/Review, 
contractor  and  government  “certification”  of  the  configuration  items  shall  be  accomplished  after 
completion  of  the  FCA  and  such  certification  shall  be  considered  an  accomplishment  of  the  SVR. 

B.  When  the  agency  responsible  for  qualification  of  the  configuration  items  at  the  contracting  agency 
judges  that  the  system  is  not  ready  for  SVR  at  the  time  of  FCA,  the  SVR  will  be  delayed  until  it  is 
determined  that  sufficient  information  on  the  system’s  qualification  is  available.  The  SVR  may  be 
delayed  up  to  the  end  of  System  testing  if  deemed  necessary. 

C.  When  a  separate  SVR  is  necessary,  the  contractor  shall  notify  the  contracting  agency  of  the 
sufficiency  of  the  configuration  items’  test  results  to  substantiate  an  SVR  and  coordinate  the  agenda 
with  the  Deputy  Director  for  Test  and  Deployment.  The  SVR  team  will  be  assembled  in  the  same 
manner  as  that  required  for  the  FCA  team.  No  duplication  of  FCA  effort  shall  occur  at  the  SVR; 
however,  the  following  additional  efforts  must  be  accomplished: 

1.  A  review  of  the  FCA  minutes  must  be  performed  and  the  SVR  shall  be  considered  as  an 
extension  of  FCA.  New/additional  qualification  data  shall  be  audited  and  reviewed  to  ensure 
qualification  of  the  configuration  items  against  the  System/Subsystem,  Software  Requirements, 
and  Interface  Requirements  Specifications. 

2.  Any  testing  accomplished  against  configuration  item  qualification  during  System  testing  shall 
be  considered. 

3.  The  contractor  shall,  after  notification  of  certification  by  the  contracting  agency,  enter  the  date 
of  system  certification  of  qualification  and  the  identity  of  the  test  reports/documentation  that 
sets  forth  the  results  of  the  associated  test(s)  in  the  configuration  item  Development  Record. 

D.  All  other  factors,  such  as  agenda,  team  organization,  review  procedures,  data  to  be  reviewed,  etc., 
shall  be  accomplished  as  delineated  in  the  FCA  and  General  Requirements  and  Procedures  sections 
of  this  standard  to  the  extent  necessary  to  accomplish  the  SVR. 

90.3  Post  Review  Action 

A.  After  conducting  the  SVR,  the  contractor  shall  publish  and  distribute  copies  of  SVR  minutes.  The 
contracting  agency  will  officially  acknowledge  the  conduct  of  the  Review  as  indicated  in  paragraph 
5.3. 
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Appendix  J  -  Manufacturing  Readiness  Review  (MRR) 

and 

Production  Readiness  Review  (PRR) 


100.  Manufacturing  Readiness  Review  (MRR)  and  Production  Readiness  Review  (PRR) 

100.1  General 

A.  Defense-unique  and/or  defense-critical  manufacturing  and  production  play  a  vital  role  in  the 
development  of  weapons  systems.  Changing  circumstances  have  significantly  influenced  defense 
manufacturing.  These  include: 

1 .  Changing  threats  to  national  security 

2.  Declining  defense  budgets 

3.  Consolidation  of  the  defense  industry 

4.  The  increasing  globalization  of  industry 

5.  The  increasing  rate  of  change  of  technology 

6.  Requirements  for  environmentally  compatible  manufacturing 

B.  Defense-unique  and/or  defense-critical  manufacturing  capabilities  are  either  broadly  applicable  to  a 
number  of  weapons  systems  or  specific  to  certain  weapons  systems,  i.e.: 

1 .  Composites  processing  and  repair 

2.  Electronics  processes 

3.  Information  technology  systems 

4.  Weapons  system  sustainment 

5.  Design,  modeling,  and  simulation 

6.  Production  processes 

C.  Defense  manufacturing  characteristics  interact  with  each  other  and  are  composed  of  the  following 
elements: 

1.  Manufacturing  accounting,  including  activity-based  accounting  and  cost-as-an- independent- 
variable  accounting 

2.  Product  design,  including  life  cycle  design,  integrated  product  and  process  development, 
three-dimensional  digital  product  models,  simulation  and  modeling,  and  rapid  prototyping 

3.  Manufacturing  processes,  including  generative  numerical  control,  adaptive  machine  control, 
predictive  process  control,  high-speed  machining,  flexible  tooling,  soft  tooling,  tool-less 
assembly,  embedded  sensors,  flip  chips,  nanotechnology,  and  biotechnology 

4.  Environmentally  compatible  manufacturing  technologies,  including  cleaning  systems, 
coatings,  and  materials  selection,  storage,  and  disposal 

5.  Business  organization,  including  teaming  among  organizations,  virtual  enterprises,  long-term 
supplier  relationships,  high-performance  organizations,  cross-functional  teams,  lean 
enterprises,  adaptive  enterprises,  agile  enterprises,  and  knowledge-based  and  learning 
enterprises 

6.  Information  and  communications  technologies,  including  electronic  commerce,  virtual  co- 
location  of  people,  data  interchange  standards,  internet  technologies,  intranet  technologies, 
browser  technologies,  intelligent  agents,  seamless  data  environments,  telecommunications, 
and  distance  learning 
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7.  Application  of  advanced  production  processes  and  practices  to  maintenance,  repair,  and 
upgrade  operations 

8.  Technology  insertion  for  new  and  existing  systems 

9.  Self-diagnostics  for  mechanical  and  electronic  systems 

10.  New  technologies  for  remanufacturing 

1 1 .  Design  and  manufacturing  methods  for  sustainment  of  weapons  systems: 

a.  Application  of  advanced  production  processes  and  practices  to  maintenance,  repair,  and 
upgrade  operations 

b.  Technology  insertion  for  new  and  existing  systems 

c.  Self-diagnostics  for  mechanical  and  electronic  systems 

d.  New  technologies  for  remanufacturing 

e.  Design  methods  that  improve  sustainment 

f.  Algorithms  for  design  tradeoffs  to  optimize  life  cycle  costs 

g.  Parametric  models  that  facilitate  design  tradeoffs  at  the  conceptual  stage 

h.  Product  databases  that  will  permit  simulation  at  various  levels  of  resolution 

12.  Widespread  application  of  commercial  off-the-shelf  (COTS)  products: 

a.  New  weapons  systems  designed  for  open  architecture  and  technology  transparency 

b.  A  central  program  and  mechanisms  to  maintain  awareness  of,  document,  and  plan  for 
new  COTS  technologies  that  can  be  incorporated  into  current  and  future  weapons 
systems,  as  well  as  to  disseminate  this  information  to  individual  program  offices 

c.  Improved  methods  of  inserting  COTS  products  in  fielded  weapons  systems 

d.  Low-cost  validation  methods  for  determining  the  adequacy  of  COTS  parts  for  military 
applications 

13.  Defense-unique  and  defense-critical  processes  with  the  broadest  range  of  applications: 

a.  Processes  that  enable  rate-transparent  production  (i.e.,  production  where  the  per-unit  cost 
is  independent  of  the  production  rate) 

b.  Processes  for  the  low-cost  fabrication  of  composite  structures 

c.  Processes  for  the  low-cost  production  and  application  of  coatings  and  structures  with  low 
observability 

d.  Defense-unique  electronic  technologies 

e.  Design,  information,  and  manufacturing  technologies  that  provide  dimensional  control  in 
the  production  of  large,  complex  parts 

100.2  Manufacturing  Readiness  Review  (MRR) 

A.  Before  commencing  manufacture  of  a  unit  or  other  contractually  designated  configuration  items  at 
the  prime  contractor,  subcontractor,  or  critical  component  supplier,  the  prime  contractor  shall 
conduct  an  MRR  to  ensure  readiness  to  build  a  quality  product  that  inherently  embodies  defense- 
unique  and/or  defense-critical  manufacturing  capabilities  characteristic  of  a  desired  defense 
contractor  identified  in  paragraph  100.1,  as  appropriate  for  the  program  under  review. 

B.  Representatives  from  the  appropriate  design,  manufacturing,  test,  parts,  material,  processes,  quality, 
and  other  responsible  organizations  shall  participate  as  a  minimum.  The  appropriate  government 
representatives  shall  be  invited  and  allowed  to  participate. 
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C.  Topics  covered  shall  include  applicable  items  identified  in  paragraph  100.1,  to  be  mutually  agreed 
to  by  the  PCO  and  the  contractor’s  program  manager  and  include  but  not  be  limited  to: 

1.  Drawing  availability  and  acceptability 

2.  Configuration  status 

3.  Producibility  of  parts  and  materials 

4.  Adequacy  of  manufacturing  processes  and  certifications 

5.  Manufacturing  planning 

6.  Current  manufacturing  trend  data 

7.  Personnel  experience  and  training  and  certifications 

8.  Tooling 

9.  Facilities 

10.  Inspection  points 

11.  Test  equipment  availability  and  calibration  status 

12.  Corrective  action  status 

13.  Manufacturing  lessons  learned  from  prior  like  hardware  builds  and  schedule 

D.  Manufacturing  and  Test  Planning:  The  contractor  shall  develop  manufacturing,  inspection,  and  test 
instructions  for  all  segments  of  the  manufacturing  cycle,  which  shall  include  flowcharts  or  other 
effective  alternative  methods  of  identifying  all  inspection  and  test  points.  The  contractor’s  quality 
assurance  organization  shall  participate  in  the  planning  and  shall  review  and  approve  the 
instructions  prior  to  release.  Instructions  shall  include  or  reference  engineering  requirements,  such 
as  drawings,  material  specifications,  process  specifications,  and  workmanship  standards,  to  assure 
that  necessary  tests  and  inspections  are  effectively  performed  to  verify  that  the  product  meets 
technical  requirements.  Test  instructions  shall  identify  the  characteristics  to  be  measured,  the 
methods  of  measurement,  and  the  point  at  which  the  test  shall  be  performed.  Any  changes  made  to 
production  processes,  equipment,  and/or  test  equipment/tooling  shall  be  documented.  Results  of 
such  changes  shall  be  assessed  as  soon  as  practicable.  The  contractor  shall  address  the  following  in 
developing  the  required  manufacturing,  inspection,  and  test  instructions: 

1.  Sequence  of  all  manufacturing,  inspection,  and  test  points  to  ensure  continuity  and 
effectiveness  of  all  operations 

2.  Inspection  and  test  performance  at  the  optimum  item  indenture  level  to  minimize  repair  or 
rework  at  higher  indenture  levels.  All  workmanship  shall  be  inspected  at  least  once  and 
preferably  twice  before  being  covered  up  by  subsequent  operations 

3.  Module-level  environmental  testing  and  bum- in  sufficiency 

4.  Cleanliness  and  contamination  control  to  include  foreign  object  control 

5.  The  adequacy  of  in-house  handling  and  packaging,  including  provisions  for  protection  of 
electrostatic  discharge-sensitive  items 

6.  Availability  and  utilization  of  applicable  drawings,  specifications,  and  standards 

7.  Clear  definition  of  acceptance  or  rejection  criteria  for  each  inspection  or  test 

8.  Thorough  monitoring  and  documentation  of  critical  items  and  their  characteristics 

9.  Visual  aids  for  inspection  and  assembly  personnel 

10.  Proper  selection,  application,  use,  and  control  of  substances,  chemicals,  shop  aids,  clothing, 
and  expendable  materials  specified  and  used  in  the  manufacturing  process  (cleaning 
materials,  adhesives,  joining  material,  solvents,  rags,  etc.) 
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11.  Test  equipment,  tooling,  jigs,  fixtures,  and  other  fabrication  equipment  to  be  utilized 

12.  Insertion  of  appropriate  mandatory  inspection  points  for  manufacturing  and  quality 
organizations 

13.  Inclusion  of  Manufacturing  Readiness  Reviews  (MRRs),  Test  Readiness  Reviews  (TRRs), 
and  Hardware  Acceptance  Reviews  (HARs)  for  units  and  other  configuration  items 

14.  Provisions  to  record  process  data,  e.g.,  start  and  stop  times,  temperatures,  torque  values,  etc. 

E.  Workmanship:  The  contractor  shall  develop  methods  that  will  assure  that  workmanship  is  adequate 
to  meet  contract  end  item  specified  requirements. 

F.  Standards:  The  contractor  shall  establish  workmanship  standards.  These  standards  can  be  part  of 
design  specifications,  drawings,  work  instructions,  or  other  readily  available  specifications  and 
standards.  These  standards  shall  be  derived  from  industry-accepted  workmanship  standards  and  also 
be  based  on  the  contractor’s  manufacturing  experience.  All  standards  shall  be  aimed  at  delivering 
the  highest- quality  and  most  reliable  hardware  possible  to  the  customer  within  the  constraints  of  the 
contract.  All  standards  shall  define  specific  detailed  acceptance  or  rejection  criteria. 

G.  Visual  Aids:  When  visual  aids  are  used  to  support  manufacturing  or  inspections,  the  contractor  shall 
identify,  maintain,  and  control  the  samples,  graphics,  or  visual  aids  that  show  acceptable 
workmanship  to  ensure  continued  usability  and  proper  configuration. 

100.3  Production  Readiness  Review  (PRR) 

A.  The  PRR  shall  be  conducted  to  provide  a  mechanism  to  assist  the  government  in  evaluation  of  the 
production  risks  and  the  contractor’s  methodology  to  manage  those  risks. 

B.  The  four  major  PRR  evaluation  elements  are: 

1.  Production  Management,  e.g.,  address  “Organizational  Assurance,”  focusing  on  how  well  the 
manufacturing  organization  is  structured  and  managed 

2.  Production  Engineering,  e.g.,  address  “Producibility  Assurance,”  focusing  on  how  the 
product  has  been  designed  for  producibility,  and  what  planning  has  been  accomplished  to 
ensure  methods,  processes,  and  test  equipment  has  been  defined  and  are  available  to  support 
manufacturing  operations 

3.  Production  Operations,  e.g.,  address  “Process  Assurance,”  focusing  on  how 
manufacturing/production  will  be  planned  and  executed 

4.  Product  Assurance,  e.g.,  address  how  quality  has  been  designed  into  the  product  and  how 
product  quality  will  be  pursued  and  verified 

C.  Production  Readiness  Risk  Calculations  Methodology:  Managing  risk  during  the  transition  from 
development  to  production  is  a  critical  task  in  the  life  cycle  of  major  system  acquisition  programs. 
For  specific  preparation  guidance  for  PRR,  the  POC/Contractor  team  may  elect  to  use  the 
Production  Readiness  Review  Tool  User’s  Guide  (AFSCR  84-2),  which  defines  the  risk  calculation 
methodology  for  the  major-element  impact  assessment  and  scoring  according  to  the  following 
criteria: 
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Impact  Assessment 

Impact  Score 

None 

0 

Insignificant 

1 

Minor 

2 

Major 

3 

Catastrophic 

4 

1 .  Risk  score  is  the  product  of  the  probability  assessment  and  the  impact.  For  example,  if  the 
probability  assessment  for  a  given  evaluation  element  is  70%,  and  the  associated  impact 
assessment  is  major,  the  calculated  risk  score  would  be  0.70  *  3  =2.1.  The  result  is  compared 
to  the  threshold  values  entered  by  the  PRR  team. 

2.  An  overall  risk  assessment  is  calculated  for  individual  evaluation  elements.  This  is  a 
controversial  approach;  some  have  argued  that  if  any  evaluation  element  within  a  sub¬ 
element  is  high  risk,  then  the  entire  sub-element  should  also  be  rated  to  be  high  risk.  The 
counter  to  this  argument  is  that  the  concept  could  itself  be  taken  to  its  extreme.  For  example, 
if  any  evaluation  element  rated  high  risk  necessitates  the  sub-element  as  high  risk,  then  the 
major-element  would  have  to  be  rated  high  risk  because  a  sub-element  is  high  risk,  and 
ultimately  the  entire  PRR  would  have  to  be  rated  high  risk  since  a  major-element  is  high  risk. 
Since  all  data  is  available  for  review,  rolling  up  the  risk  assessment  for  summary  and  out- 
brief  purposes  appears  to  be  a  reasonable  approach,  but  each  PRR  POC  and/or  Contractor 
team  should  consider  whether  they  wish  to  accept  this  approach. 

D.  Production  Processing  and  Fabrication 

1.  Certification:  The  contractor  shall  establish  a  method  to  certify  the  qualification  of  the 
machines,  equipment,  and  procedures  used  in  complex,  critical  operations.  Records  shall  be 
maintained  of  the  qualifying  tests  performed  and  the  results  of  such  tests.  Validation  prior  to 
production  shall  include  measurements  made  on  the  first  article  produced  for  a  given  design. 
Machines,  equipment,  and  procedures  shall  be  recertified  as  indicated  necessary  by  the 
results  of  quality  trends  or  when  major  process  changes  are  made  (i.e.,  such  items  as  material 
thickness,  design,  power  source,  capacity,  voltage,  or  density) 

2.  Cleanliness,  Contamination,  and  Corrosion  Control:  The  contractor  shall  review  and  identify 
the  cleanliness,  contamination,  and  corrosion  control  requirements  derived  from  hardware 
specifications  and  ensure  that  procedures  are  developed  to  adequately  protect  the  hardware 
during  manufacturing,  test,  storage,  and  transportation.  Implementation  of  controls  shall  be 
monitored  by  quality  assurance  on  a  regular  basis 

3.  Control  of  Physical  Environment:  The  contractor  shall  ensure  through  periodic  audit  that  the 
physical  environment  (such  as  temperature,  humidity,  light,  arrangement  of  work  areas,  or 
arrangement  of  machines  and  equipment)  is  controlled  to  preclude  inadvertent  damage  to 
hardware  and  to  prevent  unsafe  conditions  in  all  work  and  storage  areas 

4.  Critical  Item  Quality  Control  Requirements:  The  contractor  shall  establish  and  maintain 
appropriate  critical  item  control.  Manufacturing  shall  include  any  special  instructions  in  the 
appropriate  planning  shop  folders,  process  plans,  log  books,  and  related  documents 
controlling  the  manufacturing  and  movement  applicable  to  in-house  manufacturing. 
Components  or  materials  selected  for  preferential  treatment  shall  be  conspicuously  marked  or 
tagged  to  alert  personnel  of  special  requirements.  These  items  shall  be  segregated  or  have 
distinctively  marked  fixtures  and  locations  in  all  stock  rooms,  and  holding  and  staging  areas. 
Such  items  shall  be  regularly  and  systematically  inspected  for  condition  of  expired  time, 
cycle,  or  calendar  life.  Items  with  expired  time,  cycle,  or  calendar  life  are  to  be  identified  as 
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nonconforming  and  properly  dispositioned.  Reviews  of  selected  critical  components  shall  be 
periodically  conducted  to  verify  the  adequacy  of  work  instructions  and  standards  being  used 

5.  Critical  Item  Verification:  For  each  critical  item,  beginning  at  the  start  of  assembly  and  at 
progressive  levels  of  assembly  and  test,  the  contractor’s  quality  organization  shall  verify  that 
the  contract,  drawing,  and  specification  requirements  have  been  met  for  all  such  articles  and 
materials,  procured  or  produced.  Anomalies,  including  trends,  deviations  from  expected 
norms,  and  marginal  conditions,  shall  be  identified.  Detailed  assessment  of  the  quality  of 
these  items  and  their  manufacture  shall  include: 

a.  Identification  of  potential  design  and  layout  problems  that  could  cause  latent  defects  or 
marginal  performance 

b.  Verification  that  current  manufacturing  test  methods  and  controls  are  producing 
repeatable  products 

c.  A  review  of  manufacturing  problems,  if  any,  that  could  be  alleviated  by  additional  (or 
revision  of)  engineering  information 

d.  Verification  that  critical  parameters  are  measured  and  verified  by  applicable  test 
procedures 

e.  Decisions,  dispositions,  corrective  actions,  or  recommendations  are  evaluated  against 
appropriate  criteria  and  previous  history  data 

f.  Anomalies  noted  or  observed  during  review  are  analyzed,  evaluated,  and  dispositioned 

g.  Records  are  progressively  reviewed  and  made  part  of  the  overall  “Acceptance  Criteria” 

h.  Identification  and  resolution  of  the  differences  between  as-built  and  design 
documentation 

i.  A  review  of  failure  and  discrepancy  reports  to  identify  underlying  causes  (symptoms  or 
manifestations)  and  a  summary  of  overstress  and  induced  secondary  failures 

6.  Electrostatic  Discharge  Control  (ESD)  Program:  Procedures  shall  be  established  for  the 
surveillance  of  the  electrostatic  discharge  control  program  implementation.  This  shall  include 
identification  of  items  susceptible  to  electrostatic  discharge  and  protective  features  to  prevent 
such  damage.  As  a  minimum  this  should  include: 

a.  Design  criteria 

b.  Protected  work  areas  and  protective  clothing 

c.  Process  controls  and  workmanship  standards 

d.  Handling,  packaging,  transportation,  and  storage 

e.  Training 

f.  Marking  of  documentation,  and  hardware 

g.  Audit  plan  for  certified  ESD  workstations 

7.  Nondestructive  Evaluation:  Nondestructive  evaluation  methods  and  verification  techniques 
(and  attendant  equipment  and  facilities),  which  are  used  to  perform  quantitative 
measurements,  integrity  analysis,  and  nondestructive  testing,  shall  be  controlled  and 
integrated  into  the  contractor’s  qualification,  calibration,  certification,  and  standards 
procedures.  Nondestructive  evaluations  for  hardware  flight  configurations  shall  be  performed 
by  personnel  proficient  and  certified  in  the  scientific  field  involved 

8.  Completed  Item  Inspection  and  Test:  Prior  to  shipment  or  storage  of  a  contract  end  item,  the 
contractor  shall  review  objective  evidence  generated  during  manufacturing  and  test  of  the 
item  to  ensure  that  all  work  sequences  have  been  satisfactorily  completed  and  that  all 
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nonconformance  issues  have  been  resolved.  The  contractor  shall  maintain  records  and 
findings  of  final  review. 

9.  Statistical  Process  Control:  The  contractor’s  quality  assurance  organization  shall  participate 
in  development  of  techniques  used  to  control  process  variability.  This  should  consist  of  the 
independent  evaluations  of  design  disclosure  technical  documentation  and  manufacturing 
processes  by  qualified  personnel.  As  a  minimum,  consider  that: 

a.  Critical  quality  characteristics  are  identified,  measured,  and  verified 

b.  Data  is  collected  from  points  of  measurement 

c.  Control  limits  and  tolerance  variations  are  maintained  within  product  specification  limits 

d.  Procedures  and  methods  are  established  for  preventive  and  corrective  actions,  and 
feedback  is  provided  to  design  and  manufacturing 
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Appendix  K  -  Application  Guide  for  Tailoring  MIL-STD-1521 


110.  Application  Guide  for  Tailoring  MIL-STD-1521 

110.1  Scope 

A.  This  appendix  sets  forth  guidance  and  shall  be  used  for  the  cost-effective  application  of  the 
requirements  of  this  standard  when  this  standard  is  contractually  invoked  during  the  acquisition 
process.  This  appendix  serves  as  guidance  for  the  activity  responsible  for  the  preparation  of  contract 
requirements  and  does  not  form  a  part  of  the  contract. 

Note:  all  references  in  Appendix  K  to  MIL-STD-1521  are  intend  to  mean  this  update  to  MIL- 
STD-1521B,  TOR-2007(8583)-6414_Volume  1. 

110.2  Purpose 

A.  The  guidelines  contained  herein  implement  the  Department  of  Defense  Directive  4120.21, 
Specification  and  Standards  Application,  which  requires  all  DoD  components  to  apply  selectively 
and  tailor  military  specifications  and  standards  prior  to  their  contractual  imposition  and: 

1 .  Eliminate  inapplicable  and  unnecessary  requirements 

2.  Provide  for  adding/modifying  necessary  technical  review  and  audit  factors  not  included  in 
MIL-STD-1521 

3.  Eliminate  redundancy  and  inconsistency  with  other  contract  specifications  and  standards 

110.3  Objective 

A.  The  objective  of  this  guide  is  to  establish  the  applications  and  limitations  of  tailoring  MIL-STD- 
1521.  MIL-STD-1521  is  not  a  stand-alone  document.  It  is  dependent  upon  the  work  effort  specified 
in  the  contractual  requirements  (e.g.,  SOW,  etc.).  The  tailoring  of  specifications  shall  take  place  in 
all  phases  of  military  procurement,  but  is  especially  applicable  to  the  initial  stages  of  solicitation 
package  preparation  and  contract  negotiation.  Depending  upon  the  type  of  end  item(s)  under 
procurement,  the  reviews  and  audits  outlined  by  MIL-STD-1521  may  or  may  not  be  required  for  all 
programs. 

110.4  Considerations  for  Tailoring 

110.4.1  Relationship  to  the  Statement  of  Work 

A.  The  Program  Manager  must  keep  in  mind  that  technical  reviews  provide  visibility  into  the 
contractor’s  implementation  of  the  work  effort  required  under  the  terms  of  the  SOW  and  the 
contract  to  assure  timely  and  effective  attention  to  the  technical  inteipretation  of  contract 
requirements. 

B.  The  key  to  tailoring  MIL-STD-1521  is  to  match  the  MIL-STD-1521  requirements  against  the 
details  of  the  applicable  SOW  and  contractual  task  requirements.  It  will  become  immediately 
obvious  that  MIL-STD-1521  may  contain  technical  review  factors  that  are  not  applicable  to  the 
contract  under  consideration.  (For  example,  if  a  contract  does  not  include  computer  software,  all 
references  to  the  review  of  computer  software  materials  in  MIL-STD-1521  will  not  apply). 

C.  When  MIL-STD-1521  is  used,  then  a  task  containing  the  applicable  requirements  will  be  specified 
in  the  SOW.  Review  factors  not  set  forth  in  MIL-STD-1521  but  considered  necessary  because  of 
the  nature  of  the  particular  program  shall  be  added  in  the  SOW.  Being  subject  to  a  careful 
evaluation  process,  the  technical  review  and  audit  requirements  will  become  program  specific  rather 
than  an  all-purpose  document  to  be  continually  negotiated  during  contract  performance. 
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110.4.2  Elimination  of  Redundancy  and  Ambiguity 

A.  While  MIL-STD-1521  is  the  broad  program  document  for  technical  reviews  and  audits,  other 
standards  in  existence  also  require  technical  reviews  or  audits.  For  example,  MIL-STDs  for 
reliability,  maintainability,  system  engineering,  etc.,  can  require  reviews  and/or  audits.  Review  of 
these  aspects  of  the  design  would  also  be  required  under  MIL-STD-1521;  therefore,  if  such 
standards  are  contractually  stipulated  together  with  MIL-STD-1521,  the  SOW  shall  include  a 
provision  to  show  how  and  whether  the  technical  review  requirements  of  these  other  standards  can 
be  combined  with  technical  reviews  or  audits  in  MIL-STD-1521. 

B.  Combining  reviews  does  not  nullify  other  MIL-STD(s),  “Plans,”  etc.,  that  contain  requirements  for 
reviews  and  audits.  The  contract  shall  require  the  minimal  integrated,  comprehensive  technical 
design  review  effort  that  will  provide  the  desired  visibility  and  assurance  of  contract  compliance. 

110.4.3  Contractor  Participation  in  Tailoring 

A.  When  requiring  a  particular  review  or  audit,  it  is  important  that  the  topics  to  be  reviewed  are 
aligned  to  the  program  requirements.  Therefore,  the  offeror  shall  be  given  an  opportunity  to 
recommend  changes  and  identify  topics  or  items  he  considers  appropriate. 

B.  The  program  office  shall  request,  in  the  instructions  for  proposal  preparation,  that  the  offeror 
recommend  the  MIL-STD-1521  topics  or  items  and  their  related  details  to  be  covered  at  the  various 
reviews  or  audits  required  by  the  SOW.  This  will  allow  the  offeror  to  tailor  the  topics  or  items  and 
details  by  additions  and  deletions  for  the  particular  review  or  audit. 

C.  In  addition,  it  must  be  recognized  that  effective  tailoring  requires  several  points  of  review.  The 
requirement,  however,  for  the  review  or  audit  must  be  finalized  prior  to  contract  award. 

110.4.4  Complexity 

A.  System,  subsystem,  configuration  item  complexity,  and  type  of  program  are  central  in  determining 
both  the  need  for  and  the  number  of  such  reviews.  When  developing  a  small  noncomplex  system, 
some  reviews  may  not  be  required,  or,  if  required,  may  be  limited  in  scope.  Either  the  tailoring 
procedures  discussed  earlier  should  result  in  the  exclusion  of  MIL-STD-1521  or  in  a  tailored  MIL- 
STD-1521  that  reflects  a  limited  scope  technical  review  effort.  Conversely,  in  a  very  complex 
development,  the  review  process  will  increase  in  levels  and  numbers  of  reviews. 

B.  In  addition  to  the  above,  the  degree  of  application  is  dependent  upon  the  configuration  item’s  state 
of  development  (for  example,  new  design  vs.  commercially  available)  or  the  degree  of  any 
modifications,  if  involved.  For  example:  a  newly  developed  item  may  require  the  majority  of  the 
review  topics  or  items  and  audits,  while  a  commercially  available  configuration  item  with  the 
appropriate  documentation,  i.e.,  verified  test  results,  specifications,  drawings,  etc.,  may  require 
reviews  or  audits  limited  to  its  application  to  the  program  and  its  interfaces. 

C.  In  the  case  of  modified  designs,  one  must  consider  the  degree  and  effect  of  the  modifications. 
Reviews  and  audits  may  be  limited  to  the  modifications  and  their  interfaces. 

110.5  Scheduling  of  Technical  Reviews  and  Audits 

A.  The  schedule  for  Technical  Reviews  and  Audits  is  extremely  important.  If  they  are  conducted  too 
early,  the  item  for  review  will  not  be  adequately  defined.  Conversely,  if  the  review  is  too  late,  the 
program  commitments  could  have  been  made  erroneously,  and  correction  will  be  both  difficult  and 
costly. 

B.  For  planning  purposes,  a  good  method  for  scheduling  technical  reviews  is  to  relate  them  to  the 
documentation  requirements.  For  example,  schedule  a  PDR  after  the  hardware  Development 
Specification  or  software  architecture  and  detailed  design  and  Software  Test  Plan  are  available, 
since  the  essence  of  the  PDR  is  to  assess  the  contractor’s  approach  to  meeting  the  requirements  of 
these  documents. 

C.  Scheduling  of  audits  is  dependent  on  not  only  documentation  availability,  but  also  on 
hardware/software  availability  and  the  completion  of  the  acceptance  qualification  tests.  Table  3 
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contains  a  list  of  the  primary  documentation  associated  with  each  review  or  audit  and  the  estimated 
time  phasing. 


Table  3.  Scheduling  Reviews  and  Audits 


Review 

Time  Phase 

Primary  Documentation 

•  System  Requirement  Specification 

•  Preliminary  Operational  Concept  Document 

•  Systems  Engineering  Management  Plan 

•  System/Segment  Specification 

•  Software  Development  Plan 

•  Analysis  And  Trade  Study  Reports 

SRR 

Usually  accomplished  in 
the  Concept  Exploration 
Phase.  However,  may  be 
used  in  other  phases  when 
the  Concept  Exploration 
Phase  is  not  accomplished. 

•  System/Segment  specification 

•  Preliminary  Operational  Concept  Document  (OpsCon) 

•  Preliminary  Software  Requirements  Specifications 

•  Preliminary  Software  Interface  Requirements  Specifications 

•  Analyses,  trade  studies 

•  Drawings  Level  I  DoDDlOOOB 

•  TDP  type  and  element  per  MIL-DTL-3 1 000C 

SFR 

Usually  in  the 
Demonstration  and 
Validation  Phase 

•  Software  Requirements  Specifications 

•  Software  Architectural  Description 

•  External  Software  Interface  Requirements  Specifications 

•  Internal  Software  Interface  Requirements  Specifications 

•  Software  to  Hardware  Interface  Requirements  Specifications 

•  Engineering  Analyses 

•  Trades  Studies 

•  Technical  Performance  Metrics 

•  Software  Development  Plan 

•  Software  Master  Build  Plan 

•  Software  Test  Plan 

•  Operational  Concept  Document  (OpsCon) 

•  Initial  Software  Resources  Data  Report  (SRDR) 

•  Life  Cycle  Cost  (LCC)  Analysis 

•  Cost  As  an  Independent  Variable  (CAIV)  Studies 

SAR 

Usually  early  in  Full  Scale 
Development 

Review 

Time  Phase 

Primary  Documentation 
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•  Development  Specification 

•  Type  B  Performance  Specification 

•  Drawings  Level  I  DoDDlOOOB 

•  TDP  type  and  element  per  M1L-DTL-31000C 

•  Software  Top  Level  Design  Document 

•  Software  Test  Plan 

•  Preliminary  Computer  Resources  Integrated  Support  Document 

•  Preliminary  Computer  System  Operator’s  Manual 

•  Preliminary  Software  User’s  manual 

•  Preliminary  Computer  System  Diagnostic  Manual 

PDR 

Usually  accomplished  in 
the  Demonstration  and 
Validation  and/or  Full 

Scale  Development  Phase 

•  Draft  Product  Specification 

•  Type  C  Specification,  and  referenced  documentation 

•  Drawings  Level  I  or  II  DoDDlOOOB 

•  TDP  type  and  element  per  MIL-DTL-3 1 000C 

•  Software  Detailed  Design  Document 

•  Hardware  Interface  Design  Document 

•  Software  Test  Description 

•  Computer  Resources  Integrated  Support  Document 

•  Software  Programmer's  Manual 

•  Firmware  Support  Manual 

•  Test  Descriptions/Procedures 

•  Software  Development  Documentation 

•  Data  Base  Design  Document(s) 

CDR 

Usually  accomplished  in 
the  Full  Scale 

Development  phase 

•  Software  Test  Procedures 

•  Informal  Software  Development  Test  Results 

•  Informal  Hardware  Development  Test  Results 

TRR 

Usually  accomplished  in 
the  Full  Scale 

Development  phase 

•  Test  Plans 

FCA 

Usually  accomplished  at 
end  of  Full  Scale 
Development 

•  Test  descriptions 

•  Test  procedures 

•  Software  test  reports 

•  Computer  System  Operator’s  Manual 

•  Software  User’s  Manual 

•  Computer  System  Diagnostic  Manual 

Appendix  K 


Review  Time  Phase  Primary  Documentation 


•  Final  Part  II  Specification 

Usually  accomplished 
early  in  the  initial 
production  when  the 
developing  contractor  is 
preselected  as  the 
production  contractor. 

•  Type  C  Product  Specifications 

May  be  accomplished  at 

•  Referenced  Software  Product  Specification  Documents  and  Drawings 

PCA 

the  end  of  Full  Scale 

•  Drawings  Level  II  or  III  per  DoDDlOOOB 

Development  when  the 

•  TDP  Type  and  Element  per  MIL-DTL-31000C 

developing  contractor  is 
not  preselected  as  the 
Production  contractor.  The 

•  Software  Product  Specification 

PCA  is  repeated  with  each 
subsequent  contractor  or 
break  in  production. 

•  Version  Description  Document 

*  DoDD-lOOOB  describes  three  drawing  levels,  which  are  recommended  for  consideration  in  tailoring  by 
Appendix  K,  because  they  were  not  carried  forward  to  MIL  DTL-31000,  its  successor  document. 

110.6  Tailoring  Guidance  for  Engineering  Data  Reviews 

A.  Engineering  Data  reviews  are  conducted  as  part  of  the  formal  design  reviews/audits  in  MIL- STD- 
1521.  Use  the  Review  Checklist  for  Engineering  Data  to  help  prepare  for  and  conduct  these 
reviews  and  audits.  Note  discrepancies  on  the  Engineering  Data  Discrepancy  Sheet  (Figure  6). 
Because  reviews  and  audits  are  successively  more  detailed,  more  items  on  the  checklist  will  apply 
as  the  program  progresses.  When  all  reviews  and  audits  are  completed,  all  items  on  the  tailored 
checklist  shall  be  accomplished. 

Review  Checklist  for  Engineering  Data 

I.  The  following  questions  and  considerations  shall  be  used  prior  to  conducting  an  engineering 
data  review.  These  are  suggested  guidelines,  and  should  be  used  as  such 

II.  Pre -briefing  preparation: 

1.  Answer  these  questions: 

a.  What  is  the  purpose  of  the  review? 

b.  What  does  the  contract  require? 

c.  How  will  the  drawings  be  used? 

2.  Arrange  briefings: 

a.  The  Contractor  shall  brief  the  team  on  contractual  requirements  and  status 

b.  The  Engineering  Data  Management  Officer  (EDMO)  or  Chairperson  should  brief 
the  team  on  the  review  procedures 

c.  Discuss  corrective  action  procedures 
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III.  The  Data  Review: 

1 .  Build  the  package: 

a.  Select  sample  of  top  assembly  drawings 

b.  Look  at  Parts  List  of  the  top  assembly  or  major  subassembly  drawings 

c.  Are  other  subassembly  drawings  listed  in  the  top  parts  list? 

d.  Are  all  drawings  listed  in  the  top  parts  list  available? 

e.  Are  all  drawings  listed  in  the  subassembly  parts  list  available? 

f.  Is  manufacturing  planning  documentation  available? 

2.  Examine  the  engineering  data  for  the  following: 

a.  Is  the  drawing  legible  and  suitable  for  reproduction? 

b.  Are  processes/specifications  listed? 

c.  Look  at  notes  on  all  drawings.  Are  all  notes  understandable?  Are  notes  clear  and 
concise? 

d.  Are  peculiar  symbols,  abbreviations,  etc.,  explained? 

e.  Are  all  dimensions  and  tolerances  shown? 

f.  Is  the  material  identified? 

g.  Are  any  reports  referenced?  If  so,  are  they  supplied  in  the  package? 

h.  Are  copies  of  nongovernment  specifications  supplied  as  part  of  the  package? 

i.  Is  the  use  of  limited  rights  legends  correct  per  Defense  Acquisition  Regulation 
(DAR)  and  Federal  Acquisition  Regulation  (FAR)? 

j.  Are  control  drawings  (particularly  Source  and  Specification  Control)  properly  used 
and  marked  per  DoD-STD-100? 

k.  Are  hardness-critical  items  and  hardness-critical  process  markings  correct? 

l.  Are  electrostatic  discharge-sensitive  (ESDS)  symbology  and  cautions  included,  as 
appropriate? 

m.  Flave  changes  been  incoiporated  as  required  in  the  contract? 

n.  Are  index  and  data  lists  available  and  correct? 

o.  Is  there  a  distribution  statement  on  each  piece  of  engineering  data? 

p.  Flave  specific  marking  requirements  (MIL-STD-130)  been  defined? 

q.  Are  acceptance  test  requirements  included  on  all  subassembly/detail  drawings  for 
items  that  might  be  spared  separately  by  competitive  reprocurement? 

r.  Is  the  proper  engineering  design  information  included  for  the  level  of  drawing 
stated  in  the  contract? 

s.  Could  a  military  standard  or  specification  be  used  in  lieu  of  drawings? 

t.  Are  applicable  security  classifications  marked  correctly? 

u.  Are  the  contractual  requirements  adequate? 

v.  Does  the  drawing  package  appear  to  be  adequate  to  support  the  intended  end  use 
(i.e.,  logistics  support,  competitive  reprocurement,  etc.)? 

3.  Record  all  deficiencies/discrepancies  on  the  Engineering  Data  Discrepancy  Sheet  (see 
Figure  6)  in  sufficient  detail  to  completely  define  the  problem  and  action  required 
for  compliance 
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B.  Although  the  time  frame  for  reviews  and  audits  is  suggested  above,  it  may  vary,  depending  on  the 
particular  program.  The  schedule  for  each  review  or  audit  may  be  requested  from  the  offeror  as  part 
of  his  proposal,  or  as  part  of  the  systems  engineering  management  plan  (which  can  be  part  of  the 
proposal). 

At  the  end  of  the  review,  the  EDMO  (or  Review  Team  Chief)  collects  all  discrepancy  sheets,  signs  them, 
and  determines  appropriate  disposition.  After  resolution  of  discrepancies,  the  sheets  will  be  filed  in  the 
Engineering  Data  Files. 
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Sheet  of 
(Program  Name) 

ENGINEERING  DATA  DISCREPANCY  SHEET 
(To  be  used  with  the  Review  Checklist) 

Prime  and  Subcontractor/Vendor  Name: 

Type  of  Review: 

Reviewer’s  Name  Drawing/Document  Number  Rev.  Date 

Discrepancies 

Action  Required/Compliance  Due  Date: 

Program  Office  EDMO  (or  Team  Chief)  Signature: _ 

Air  Logistics  EDMO  Signature:  _ 

Action  Agency: 

□  Contractor  Q  Program  Office 

I  I  Contract  Administration  Office  O  Other 

This  block  to  be  used  by  Action  Agency. 

Discrepancies  corrected  by: _ _ 

_ Signature _ Date _ 

Figure  6.  Engineering  Data  Discrepancy  Sheet 

After  resolution,  return  the  Engineering  Data  Discrepancy  Sheet  to  the  Program  Office  EDMO. 
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Part  I  -  Definitions 


Analysis  of  Alternatives  (AoA) 

AoA  is  the  evaluation  process  of  the  operational  effectiveness,  operational  suitability,  and  estimated  costs 
and  risks  of  alternative  system  concepts  to  meet  a  mission  capability.  The  analysis  assesses  the 
advantages  and  disadvantages  of  alternatives  being  considered  to  satisfy  capabilities,  including  the 
sensitivity  of  each  alternative  to  possible  changes  in  key  assumptions  or  variables. 

Architecture 

Architecture  is  a  structure  of  components,  their  relationship,  and  the  principles  and  guidelines  governing 
their  design  and  evolution  over  time;  or, 

Architecture  is  a  structure  or  organization  that  shows  the  elements  and  their  relationship  for  a  set  of 
requirements  or  a  system  concept  or  both;  or, 

Architecture  is  a  high-level  property  or  attribute  of  a  system,  such  as  openness  or  interoperability  or  a 
standard  containing  the  aforementioned. 

Architecture  Views 

Architecture  views  are  the  standardization  of  the  format  and  content  of  architectural  products.  The 
Department  of  Defense  (DoD)  Architecture  Framework  (DoDAF)  defined  architecture  views  are: 

a.  All  (AV) 

b.  Operational  (OV) 

c.  Systems  (SV) 

d.  Technical  (TV) 

The  operational  concept  (OPCON)  drives  the  OV.  The  OV  in  turn  drives  the  SV  to  identify  shortfalls  and 
system  requirements,  and  the  SV  requirements  drive  the  TV  to  address  a  common  set  of  applicable 
standards,  with  the  AV  providing  the  overarching  aspects  of  an  architecture  that  relates  to  all  three  of  the 
OV,  SV,  and  TV.  The  architecture  description  provides  the  explicit  linkages  among  various  views:  i.e., 
“interoperability”  is  a  typical  architecture  focus  that  demonstrates  the  criticality  of  developing  these 
relationships  among  the  various  views. 

Alternative  Systems  Review  (ASR) 

The  ASR  is  conducted  to  ensure  that  the  resulting  set  of  requirements  agrees  with  the  customers’  needs 
and  expectations  and  that  the  system  under  review  can  proceed  into  the  Technology  Development  phase. 
The  ASR  assesses  the  alternative  systems  that  have  been  evaluated  during  the  Concept  Refinement  phase, 
and  ensures  that  the  Technology  Development  plan  is  consistent  with  the  preferred  system  solution  and  is 
adequately  resourced  to  reduce  System  Development  and  Demonstration  entry  risk  to  an  acceptable 
level.The  ASR  ensures  that  the  preferred  system  alternative  is  cost  effective,  affordable,  operationally 
effective,  and  suitable,  and  can  be  developed  to  provide  a  timely  solution  to  a  need  at  an  acceptable  level 
of  risk. 

It  is  held  well  in  advance  of  Milestone  A  to  allow  time  for  issue  resolution  and  proper  executive-level 
concurrence  on  process  and  results. 

Audits  and  Reviews 

Audits  and  reviews  are  scheduled  events,  usually  conducted  at  any  time  after  PDR,  independently  or  as 
part  of  other  reviews  to  validate  that  the  development  of  a  configuration  item(s)  satisfies  the  Form,  Fit, 
Functional  (FFF),  and  performance  objectives  of  the  Program.  These  audits  and  reviews  can  encompass 
but  are  not  limited  to  the  following:  Functional  Configuration  Audit  (FCA) 

a.  Physical  Configuration  Audit  (PCA) 

b.  System  Verification  Review  (SVR) 
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c.  Ship  Readiness  Reviews,  i.e.,  Consent  to  Ship  (CTS) 

d.  Mission,  Flight,  and/or  Launch  Readiness  Reviews  (M/F/LRR) 

Baseline  (B/L  or  BL) 

Baseline  is  a  specification  or  product  that  has  been  formally  reviewed  and  agreed  upon,  that  thereafter 
serves  as  the  basis  for  further  development.  Baselines  can  be  changed  only  through  formal  change  control 
procedures  that  require  contracting  agency  review  and  approval.  The  initially  documented  and  validated 
system-level  requirements  and  constraints,  their  allocation  or  assignment  to  the  next  level,  and  all  changes 
thereto  are  approved  in  accordance  with  the  contract  SOW.  Typically,  baseline  requirements  are  initially 
approved  at  the  System  Functional  Review  (SFR)  or  similar  event. 

Specifically,  e.g.,  technical  baselines  are  either: 

a.  Functional: 

(1)  Functional  Baseline  (FBL)  is  validated  and  approved  in  conjunction  with  physical  and 
performance  system-level  requirements  and  associated  design  constraints 

(2)  Allocated: 

(3)  Allocated  Baseline  (ABL)  is  the  physical  hierarchy  of  an  approved  allocated 
configuration 

(4)  The  initially  documented,  validated,  and  approved  design-to  functional  and 
performance  requirements  and  design  constraints  for  each  system  product  in  the 
hierarchy  and  all  changes  thereto  approved  in  accordance  with  the  contract 

(5)  The  separable  documentation  identifying  all  design-to  requirements  and  constraints  for 
each  component  or  computer  software  item  and  each  separately  integrated  grouping  of 
components  and/or  computer  software  items,  or 

b.  Product  (End  Item/C onfiguration)  such  as: 

(1)  Build-to  requirements  for  each  physical  element  to  be  manufactured 

(2)  Software  code  for  each  software  element  that  has  been  separately  designed  or  tested 

(3)  Buy-to  requirements  for  each  physical  element,  part,  or  material  to  be  procured 

(4)  The  initially  documented  and  approved  update  to  the  design  release  baseline  for  one  or 
more  end  items  (Els)  after  confirmation  that: 

a)  The  El  design  satisfies  all  performance  and  functional  requirements  and  constraints 
in  the  current  allocated  and  design  release  baselines 

b)  The  as-built,  as-coded,  or  as-integrated  product  accurately  reflects  the  baseline 

c)  The  hardware  and  software,  readiness  for  continued  production,  acceptance 
verification,  deployment,  training,  operations,  support,  and  disposal  and  all 
subsequent  changes 

Budget/Cost 

Budget  and  Cost  terminology  guidance  can  be  found  in  DoD  5000.4M. 

Critical  Design  Review  (CDR) 

The  CDR  is  a  multidisciplinary  technical  review,  conducted  for  each  configuration  item  when  detailed 
design  is  essentially  complete,  with  the  objective  of  ensuring  that  the  detailed  design  of  the  configuration 
item  or  system  under  review  can  proceed  into  fabrication,  system  integration,  demonstration,  and  test,  and 
can  meet  the  stated  performance  and  engineering  specialty  requirements  of  the  configuration  item  (Cl) 
development  specifications  within  cost  (program  budget),  schedule  (program  schedule),  risk,  and  other 
system  constraints: 
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a.  The  CDR  establishes  the  detailed  design  compatibility  among  the  configuration  item  and 
other  items  of  equipment,  facilities,  computer  software  and  personnel 

b.  The  CDR  assesses  configuration  item  risk  areas  (on  a  technical,  cost,  and  schedule  basis) 

c.  The  CDR  assesses  the  results  of  the  producibility  analyses 

d.  The  CDR  focuses  on  the  determination  of  the  acceptability  of  the  detailed  design, 
performance,  and  test  characteristics  of  the  design  solution,  and  on  the  adequacy  of  the 
operation  and  support  documents 

e.  The  CDR  assesses  the  system  final  design  as  captured  in  product  specifications  for  each 
configuration  item  in  the  system  (product  baseline),  and  ensures  that  each  product  in  the 
product  baseline  has  been  captured  in  the  detailed  design  documentation,  e.g.,  product 
specifications  for: 

(1)  Hardware,  to  enable  the  fabrication  of  configuration  items,  and  may  include  production 
drawings 

(2)  Software  (e.g.,  software  architecture  and  detailed  design  of  a  Software  Configuration 
Item  (SWCI)),  to  the  extent  specified  in  the  SDP  based  on  the  selected  life  cycle 
model(s) 

f.  For  complex  systems,  the  contractor  may  conduct  a  CDR  for  each  subsystem  or 
configuration  item.  These  individual  reviews  would  lead  to  an  overall  system  CDR.  When 
individual  reviews  have  been  conducted,  the  emphasis  of  the  overall  system  CDR  shall 
focus  on  configuration  item  functional  and  physical  interface  design,  as  well  as  overall 
system  detailed  design  requirements 

g.  The  System  CDR  determines  whether  the  hardware,  human,  and  software  final  detailed 
designs  are  complete,  and  whether  the  contractor  is  prepared  to  start  system  fabrication, 
demonstration,  and  test 

Configuration  Item  (Cl) 

A  configuration  item  is  an  item  that  satisfies  a  documented  set  of  requirements  and  includes  any  item 
required  for  logistics  support  or  designated  for  separate  procurement.  Configuration  items  may  consist  of 
hardware  or  software,  or  an  aggregation  of  both  that  is  designated  by  the  contracting  agency  as  an  end 
item  (El)  or  proposed  by  the  contractor  for  development/functional  end  use  and  is  designated  for 
individual  configuration  management. 

a.  A  configuration  item  is  any  hardware,  software,  or  combination  of  both  that  satisfies  an  end 
use  function  and  is  designated  for  separate  configuration  management.  Configuration  items 
are  typically  referred  to  by  an  alphanumeric  identifier  that  also  serves  as  the  unchanging 
base  for  the  assignment  of  serial  numbers  to  uniquely  identify  individual  units  of  the  Cl 
(See  also:  Product-Tracking  Base-Identifier) 

b.  The  terms  “Cl”  and  “Product”  are  identified  as  aliases  in  ANSI/EIA  649  and  are  used 
interchangeably  within  MIL-HDBK-61A 

c.  A  configuration  item  is  an  aggregation  of  hardware,  firmware,  computer  software,  or  any  of 
their  discrete  portions  that  satisfies  an  end  use  function,  and  is  designated  by  the 
Government  for  separate  configuration  management.  CIs  may  vary  widely  in  complexity, 
size,  and  type,  from  an  aircraft,  electronic,  or  ship  system  to  a  test  meter  or  round  of 
ammunition.  Any  item  required  for  Logistics  Support  (LS)  and  designated  for  separate 
procurement  is  a  Cl  [Glossary  of  Defense  Acquisition  &  Terms,  2005] 
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Concept  of  Operations  (CONOPS) 

CONOPS  is  a  high-level  concept  whose  purpose  is  to  describe  a  problem  that  combatant  commanders 
may  encounter,  identify  needs,  achieve  objectives,  capabilities,  or  desired  effects,  and  sequenced  actions 
that  describe  their  employment. 

a.  CONOPS  are  typically  developed  by  the  operator/user  with  support  from  the  acquisition 
agency  planner  or  System  Program  Office  (SPO),  for  employing  and  supporting  a 
capability  or  system  concept.  CONOPS  are  used  in  System  Technical  Requirements 
Analysis  to  identify  system  functional  requirements  and  design  constraints 

Commercial  off  the  Shelf  (COTS) 

COTS  is  a  system  product  that  is  available  in  the  commercial  marketplace  that  does  not  require  unique 
contracting  agent  modifications  or  maintenance  over  its  life  cycle  to  meet  the  requirements. 

Effectivity  (E) 

Effectivity  is  a  designation,  defining  the  point  in  time,  an  event,  a  capability  (e.g.,  system,  segment,  SoS, 
IOC)  or  a  product  range  (e.g.,  serial,  lot  number,  model,  date)  at  which  changes  or  variances  thereof  are  to 
be  incorporated. 

Engineering  Data 

Engineering  data  is  the  recorded  scientific  or  technical  information  (regardless  of  the  form  or  method  of 
recording),  including  computer  software  documentation  that  defines  and  documents  an  engineering  design 
or  product  configuration  (sufficient  to  allow  duplication  of  the  original  items)  and  is  used  to  support 
production,  engineering,  and  logistics  activities  that  may,  e.g.,: 

Include: 

a.  All  final  plans,  procedures,  reports,  and  documentation  pertaining  to  systems,  subsystems, 
computer  and  computer  resource  programs,  component  engineering,  operational  testing, 
human  factors,  reliability,  availability,  and  maintainability,  and  other  engineering  analysis, 
etc. 

b.  Technical  data  package  per  MIL-DTL-31000  (reprocurement  package),  that  includes  all 
engineering  drawings  per  ASME  Y14.100M,  associated  lists,  process  descriptions,  and 
other  accompanying  documents,  manufacturer  specifications,  manufacturing  planning 
documentation  and  standards  defining  physical  geometry,  material  composition,  and 
performance  procedures 

c.  Other  information  prepared  by  a  design  activity,  relating  to  the  design,  manufacture, 
procurement,  test,  or  inspection  of  hardware/software  items  or  services 

Exclude: 

a.  Computer  software  or  financial,  administrative,  cost  or  pricing,  or  management  data  or 
other  information  incidental  to  contract  administration 

Flight  Readiness  Review  (FRR) 

The  Flight  Readiness  Review  (FRR)  is  a  multidisciplinary  product  and  process  assessment  that  is 
performed  to  ensure  that  the  system  under  review  can  proceed  into  flight  test  with  airworthiness  standards 
met,  objectives  clearly  stated,  flight  test  data  requirements  clearly  identified,  and  an  acceptable  risk 
management  plan  defined  and  approved. 

The  FRR  is  convened  to: 

a.  Ensure  that  proper  coordination  has  occurred  between  engineering  and  flight 

b.  Ensure  that  all  applicable  disciplines  understand  and  concur  with  the  scope  of  effort  that 
has  been  identified 
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c.  Assess  the  adequacy  of  how  this  effort  will  be  executed  to  derive  the  data  necessary  to 
satisfy  airworthiness  and  test  and  evaluation  requirements 

d.  Assess  the  sufficiency  and  appropriate  level  of  detail  for  each  configuration  to  be  evaluated 
within  the  flight  test  effort 

Function 

A  function  is  a  task  to  be  performed  to  achieve  a  required  outcome  or  satisfy  an  operational  need. 

Functional  (Analysis,  Allocation,  and  Architecture) 

Functional  Analysis  and  Allocation 

a.  The  determination  of  the  top-level  functions  that  are  needed  to  accomplish  the  primary 
system  functions  over  the  life  of  the  system,  their  relationship,  and  their  decomposition  to 
sub-functions  to  the  point  that  each  sub-function  or  set  of  sub-functions  can  be  related  to 
one  and  only  one  physical  element  in  the  allocated  baseline,  the  allocation  of  the  top-level 
requirements  and  constraints  in  the  requirements  baseline  to  determine  how  well  each 
function  and  sub-function  must  be  performed,  and  the  capture  of  the  aggregate  in  a 
functional  architecture 

Functional  Architecture 

a.  The  result  of  functional  analysis  and  allocation 

b.  The  hierarchical  arrangement  of  functions,  their  decomposition  into  sub-functions,  the 
associated  timelines,  and  the  allocation  of  the  performance  requirements  and  constraints  in 
the  requirements  baseline  to  the  functions  and  sub-functions 

c.  Interfaces  between  the  functional  elements 

Incremental  Development 

a.  Incremental  development  is  a  life  cycle  model  where  all  desired  capabilities  are  identified; 
all  end-state  requirements  are  known  and  understood  up  front  but  are  implemented  in 
pieces,  or  increments.  Since  all  requirements  are  known  in  the  beginning,  all  increments 
can  be  planned  in  advance;  their  implementation  sequencing  order  and  overlaps  would  be 
determined  at  the  beginning  of  increment  development 

Information  Exchange  Requirements  (IERs) 

a.  IERs  are  requirements  that  define  the  interoperability  KPP  threshold  and  objective  values 
documented  in  Capability  Development  Documents  (CDDs),  Capability  Production 
Documents  (CPDs),  and  Capstone  Requirements  Documents  (CRDs) 

b.  The  IERs  document  both  the  information  needs  required  by  the  system  under  consideration 
and  the  needs  of  other  supported  systems 

c.  The  IERs  document  all  communication  and  computing  requirements  for  command,  control, 
communications,  computers,  and  intelligence  (C4I)  of  the  proposed  system 

Initial  Technical  Review  (FI  R) 

The  ITR  is  a  multidisciplinary  technical  review  that  supports  the  program’s  initial  Program  Objective 

Memorandum  (POM)  submission. 

a.  The  ITR  assesses  the  envisioned  requirements  and  conceptual  approach  of  the  proposed 
program  and  verifies  that  the  requisite  research,  development,  test,  engineering,  logistic, 
and  programmatic  bases  for  the  project  reflect  the  complete  spectrum  of  technical 
challenges  and  risks 
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b.  This  review  ensures  that  a  program’s  technical  baseline  is  sufficiently  rigorous  to  support  a 
valid  cost  estimate  (with  acceptable  cost  risk)  and  enable  an  independent  assessment  of  that 
estimate  by  cost,  technical,  and  program  management  subject  matter  experts 

c.  The  ITR  is  held  well  in  advance  of  the  actual  cost  estimate  submission  to  allow  time  for 
issue  resolution  and  proper  executive-level  concurrence  on  process  and  results.  The  ITR 
may  be  conducted  at  any  time  prior  to  the  ASR 

Interoperability 

The  ability  of  systems,  units,  or  forces  to  provide  data,  information,  materiel,  and  services  to  and  accept 
the  same  from  other  systems,  units,  or  forces  and  to  use  the  data,  information,  materiel,  and  services  so 
exchanged  to  enable  them  to  operate  effectively  together.  Information  technology  and  National  Security 
Systems  interoperability  includes  both  the  technical  exchange  of  information  and  the  end-to-end 
operational  effectiveness  of  that  exchanged  information  as  required  for  mission  accomplishment. 

Key  Performance  Parameter  (KPP) 

KPP  is  the  minimum  attribute  or  characteristic  considered  most  essential  for  an  effective  military 
capability. 

a.  KPPs  are  El  capabilities  that  must  be  met 

b.  KPPs  are  critical  performance  requirements  for  which  the  user  is  willing  to  consider 
cancellation  of  the  program  if  a  requirement  is  not  met 

Life  Cycle 

Life  cycle  is  the  scope  of  a  system  or  upgrade  evolution  beginning  with  the  determination  of  a  mission 
need  or  identification  of  a  system  deficiency  through  all  subsequent  phases  through  disposal  of  the 
system. 

Manufacturing 

The  term  “manufacturing”  covers  a  broad  set  of  functional  tasks  required  to  harness  all  the  elements 
needed  to  make  a  product.  Included  are  such  wide-ranging  topics  as  the  National  Technology  and 
Industrial  Base  (NTIB)  capabilities  to  support  the  program,  influencing  the  design  for  cost-effective 
manufacturing,  the  people  and  skills  needed,  the  selection  of  materials,  appropriate  methods  of 
production,  capable  machinery,  scheduling,  measurements,  and  quality  assurance  management  systems. 
Manufacturing  requires  the  support  of  functional  specialties  from  a  diverse  set  of  organizations,  including 
matrix-assigned  manufacturing  managers,  other  program  office  functionals,  contract  administration 
services  personnel,  laboratories,  contractors,  and  commodity  staff  as  well  as  depot  personnel. 

Manufacturing  Readiness  Levels  (MRLs) 

Manufacturing  Readiness  Levels  (MRLs)  are  measures  used  to  assess  manufacturing  readiness  and 
producibility  maturity  from  a  manufacturing  perspective.  MRLs  provide  a  common  understanding  of  the 
relative  maturity  (and  attendant  risks)  associated  with  manufacturing  technologies,  products,  and 
processes  being  considered  to  meet  requirements,  identifying  and  mitigating  manufacturing-associated 
risks  in  acquisition  programs,  reducing  manufacturing  risk  and  demonstrating  producibility  prior  to  full- 
rate  production. 
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Table  4.  DoD  MRL  Definitions 


MRL 

Definitions 

■ 

^evel 

1-3 

Manufacturing  concepts  identified. 

Level  4 

System,  component,  or  item  validation  in  a  laboratory  environment. 

Level  5 

System,  component,  or  item  validation  in  initial  relevant  environment.  Engineering 
application/breadboard,  brassboard  development. 

Level  6 

System,  component,  or  item  in  prototype  demonstration  beyond  breadboard, 
brassboard  development. 

Level  7 

System,  component,  or  item  in  advanced  development. 

Level  8 

System,  component,  or  item  in  advanced  development.  Ready  for  LRIP. 

Level  9 

System,  component,  or  item  previously  produced  or  in  production,  or  the  system, 
component,  or  item  is  in  LRIP.  Ready  for  Full  Rate  Production  (FRP). 

Level  10 

System,  component,  or  item  previously  produced  or  in  production,  or  the  system, 
component,  or  item  is  in  FRP. 

Table  5.  DoD  MRL  Descriptions 


MRL 

Description 

1 

^evel 

1-3 

Identification  of  current  manufacturing  concepts  or  producibility  needs  based  on 
laboratory  studies. 

Level  4 

This  is  the  lowest  level  of  production  readiness.  Technologies  must  have  matured  to 
at  least  TRL  4.  At  this  point,  few  requirements  have  been  validated,  and  there  are 
large  numbers  of  engineering/design  changes.  Component  physical  and  functional 
interfaces  have  not  been  defined.  Materials,  machines,  and  tooling  have  been 
demonstrated  in  a  laboratory  environment.  Inspection  and  test  equipment  have  been 
demonstrated  in  a  laboratory  environment.  Manufacturing  cost  drivers  are  identified. 
Producibility  assessments  have  been  initiated. 

Level  5 

Technologies  must  have  matured  to  at  least  TRL  5.  At  this  point,  not  all 
requirements  have  been  validated,  and  there  are  significant  engineering/design 
changes.  Component  physical  and  functional  interfaces  have  not  been  defined. 

Materials,  machines,  and  tooling  have  been  demonstrated  in  a  relevant 
manufacturing  environment,  but  most  manufacturing  processes  and  procedures  are  in 
development  (or  ManTech  initiatives  are  ongoing).  Inspection  and  test  equipment 
have  been  demonstrated  in  a  laboratory  environment.  Production  cost  drivers/goals 
are  analyzed.  System-level  DTC  goals  are  set.  Producibility  assessments  ongoing. 
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Level  6 

During  the  prototype  demonstration  phase,  requirements  are  validated  and  defined. 
However,  there  are  still  many  engineering/design  changes,  and  physical  and 
functional  interfaces  are  not  yet  fully  defined.  Technologies  must  have  matured  to  at 
least  TRL  6.  Raw  materials  are  initially  demonstrated  in  relevant  manufacturing 
environment.  Similar  processes  and  procedures  have  been  demonstrated  in  a  relevant 
manufacturing  environment.  At  this  point,  there  are  likely  major  investments 
required  for  machines  and  tooling.  Inspection  and  test  equipment  should  be  under 
development.  Producibility  risk  assessments  are  ongoing  and  trade  studies 
conducted.  A  production  Cost  Reduction  Plan  is  developed.  Production  goals  are 
met. 

Level  7 

Technologies  must  have  matured  to  at  least  TRL  7.  At  this  point,  engineering/design 
changes  should  decrease.  Physical  and  functional  interfaces  should  be  clearly 
defined.  All  raw  materials  are  in  production  and  available  to  meet  the  planned  Low 

Rate  Initial  Production  (LRIP)  schedule.  Pilot  line  manufacturing  processes  and 
procedures  are  set  up  and  under  test.  Processes  and  procedures  not  yet  proven  or 
under  control.  During  this  phase,  initial  producibility  improvements  should  be  under 
way.  DTC  estimates  are  less  than  125  percent  of  goals.  Detailed  production 
estimates  are  established. 

Level  8 

Technologies  must  have  matured  to  at  least  TRL  8.  At  this  point,  engineering/design 
changes  should  decrease  significantly.  Physical  and  functional  interfaces  should  be 
clearly  defined.  All  raw  materials  are  in  production  and  available  to  meet  planned 

LRIP  schedule.  Manufacturing  processes  and  procedures  have  been  proven  on  the 
pilot  line  and  are  under  control  and  ready  for  LRIP.  During  this  phase,  initial 
producibility  risk  assessments  should  be  completed.  Production  cost  estimates  meet 

DTC  goals. 

Level  9 

During  LRIP,  all  systems  engineering/design  requirements  should  be  met,  and  there 
should  be  only  minimal  system  engineering/design  changes.  Technologies  must  have 
matured  to  at  least  TRL  9.  Materials  are  in  production  and  available  to  meet  planned 
production  schedules.  Manufacturing  processes  and  procedures  are  established  and 
controlled  in  production  to  three-sigma  or  some  other  appropriate  quality  level. 

Machines,  tooling,  and  inspection  and  test  equipment  deliver  three-sigma  or  some 
other  appropriate  quality  level  in  production.  Production  risk  monitoring  is  ongoing. 

LRIP  actual  costs  meet  estimates. 

Level  10 

The  highest  level  of  production  readiness.  Minimal  engineering/design  changes. 

System,  component,  or  item  is  in  production  or  has  been  produced  and  meets  all 
engineering,  performance,  quality,  and  reliability  requirements.  All  materials, 
manufacturing  processes  and  procedures,  and  inspection  and  test  equipment  are 
controlled  in  production  to  six-sigma  or  some  other  appropriate  quality  level  in 
production.  A  proven,  affordable  product  is  able  to  meet  the  required  schedule. 

Production  actual  costs  meet  estimates. 
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Net-Centric 

Relating  to  or  representing  the  attributes  of  a  net-centric  environment.  A  net-centric  environment  is  a 
robust,  globally  interconnected  network  environment  (including  infrastructure,  systems,  processes,  and 
people)  in  which  data  is  shared  in  a  timely  manner  and  seamlessly  among  users,  applications,  and 
platforms.  A  net-centric  environment  enables  substantially  improved  military  situational  awareness  and 
significantly  shortened  decision-making  cycles. 

Net-Ready  Key  Performance  Parameter  (NR-KPP) 

The  NR-KPP  assesses  information  needs,  information  timeliness,  information  assurance,  and  net-ready 
attributes  required  for  both  the  technical  exchange  of  information  and  the  end-to-end  operational 
effectiveness  of  that  exchange.  The  NR-KPP  consists  of  measurable  and  testable  characteristics  and/or 
performance  metrics  required  for  timely,  accurate,  and  complete  exchange  and  use  of  information  to 
satisfy  information  needs  for  a  given  capability. 

Requirements  Allocation  Document  (RAD) 

The  RAD  is  the  documentation  of  the  relationships  between  the  elements  of  a  system’s  architecture  (i.e., 
SoS,  segments,  elements,  subsystems,  and  lower-level  HW  and  SW  configuration  items  and  units)  and  the 
requirements  (interface,  functional,  quality,  test,  etc.)  as  well  as  program/system  specific/unique 
conditions  otherwise  not  stipulated  in  the  product’s  technical  requirements  document  or  user  requirement. 

The  RAD  provides  the  evidence  that  the  following  items  are  observed: 

a.  All  requirements  and  marginal  conditions  are  addressed  by  every  element  of  the  system 
architecture 

b.  Every  requirement  is  allocated  to  at  least  one  element  of  the  technical  architecture 

c.  Each  requirement  is  flowed  down  to  the  lowest  element 

d.  Every  requirement  is  verifiable  through  a  defined  methodology  with  appropriate  success 
criteria  and  documented  by  a  Requirements  Verification  Traceability  Matrix  (RVTM) 

Each  allocated  requirement  has  a  parent/child  relationship  that  is  bidirectionally  traceable  and 
documented  by  a  Requirements  Traceability  Matrix  (RTM). 

Requirements 

Requirements  are  unambiguous  statements  that  identify  a  system,  product,  or  process  characteristic  or 
constraint  that  can  be  verified  and  are  deemed  necessary  for  stakeholder  acceptance. 

Requirements  are  conditions  or  capabilities  needed  by  a  user  to  solve  a  problem  or  achieve  an  objective 
that  must  be  met  or  possessed  by  a  system  or  system  component  to  satisfy  a  contract,  standard, 
specification,  or  other  formally  imposed  documents,  or  a  documented  representation  of  a  condition  or 
capability  at  various  stages  of  the  development  process. 

Requirements  are  initially  approved  at  the  System  Requirements  Review  (SRR)  or  similar  event,  and 
can  encompass  but  are  not  limited  to  the  following  types: 

a.  Acceptability  (Requirements) 

Acceptability  requirements  define  a  system  that  satisfies  all  user  capabilities  and  requirements. 
All  user  requirements  trace  to  system-  and  lower-level  requirements 

b.  Analysis  (Requirements) 

Requirements  analysis  is  the  determination  of  a  complete  and  verifiable  system  functional  and 
technical  performance  requirements  and  design  constraints,  based  on  analyses  of  the  needed 
operational  capabilities,  requirements,  objectives  (or  goals),  measures  of  effectiveness;  missions; 
projected  utilization  environments;  DoD  policies  and  practices;  public  law;  and  the  balance 
between  capabilities  to  be  provided  and  the  evolutionary  growth  potential,  on  the  one  hand,  and 
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cost,  schedule,  and  risk,  on  the  other  hand.  The  results  of  requirements  analyses  are  documented 
in  the  requirements  baseline. 

c.  Baseline  (Requirements) 

Baseline  requirements  are  the  documented,  validated,  and  approved  system-level  (top-level) 
verifiable  and  allocable  functional  and  performance  technical  requirements  and  design 
constraints,  their  allocation  or  assignment  to  the  next  level  (and  lower  levels  if  necessary  to 
capture  the  systems  engineering  foundation  for  the  program),  and  all  changes  thereto  approved  in 
accordance  with  the  contract 

d.  Constraints  (Requirement) 

Constraints  are  requirements  that  form  boundaries  within  which  other  requirements  must  be 
allocated  or  derived  and  system  products  must  be  designed.  The  constraints  may  be  externally 
imposed  by  an  interface  with  another  system  or  result  from  decisions  internal  to  the  program  or 
contract.  Constraints  are  often  driven  by  federal  law,  DOD  policy  and  direction,  and  or  required 
standards  and  specifications 

e.  Derived  (Requirements) 

Derived  requirements  are  requirements  that  are  not  explicitly  stated  in  the  capability  need  that  are 
inferred  from  the  nature  of  the  proposed  solution;  the  applicable  verification,  rework,  storage, 
transportation,  operating,  and  support  environments;  policy;  law;  best  engineering  practice;  or 
some  combination  of  the  above 

f.  Design-to  (Requirements) 

Design-to  requirements  are  the  allocated  and  derived  verifiable  technical  requirements  and  design 
constraints  to  which  the  design  of  a  system  product,  including  hardware,  software,  processes, 
data,  or  new  or  modified  contracting  agent  facilities  is  to  comply 

g.  Development  (Requirements) 

Requirements  Development  is  the  process  of  taking  all  input  from  relevant  stakeholders  and 
translating  it  into  technical  form,  fit  (suitability),  function,  and  performance  (FFF&P) 
requirements 

h.  Functional  (Requirement) 

A  functional  requirement  is  a  need  or  a  capability  that  must  be  provided  by  a  system  or  end 
product/El 

i.  Interface  (Requirements) 

(1)  Physical 

(2)  Functional 

(3)  Internal 

(4)  External 

(5)  Programs 

j.  Performance  (Requirements) 

Performance  requirements  are  statements  of  the  extent  to  which  a  function  must  be  executed, 
generally  measured  in  such  terms  as  quantity,  quality,  coverage,  timeliness,  or  readiness.  See 
Functional  Requirement 
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k.  Reference  (Requirements) 

A  reference  requirement  is  a  higher-level  requirement  and/or  an  analysis,  test,  or  other 
justification  for  a  requirement,  requirement  allocation,  or  other  baseline  or  functional  architecture 
element 

Risk 

Risk  is  a  potential  problem  or  uncertainty  that  may  occur  in  the  future  that  could  result  in  the  inability  to 
achieve  an  objective  within  defined  parameters  or  constraints  such  as  program  goals  and  objectives  within 
defined  cost,  schedule,  and  technical  performance. 

Risk  has  two  components: 

a.  The  probability /likelihood  of  failure  to  achieve  a  particular  outcome 

b.  The  consequences/impacts  of  failure  to  achieve  that  outcome 

Risk  Management  Process 

Risk  management  is  a  continuous  process  that  is  accomplished  throughout  the  life  cycle  of  a  system.  It  is 
an  organized  methodology  for  continuously  identifying  and  measuring  the  unknowns;  developing 
mitigation  options;  selecting,  planning,  and  implementing  appropriate  risk  mitigation;  and  tracking  the 
implementation  to  ensure  successful  risk  reduction.  Effective  risk  management  depends  on  risk 
management  planning;  early  identification  and  analyses  of  risks;  early  implementation  of  corrective 
actions;  continuous  monitoring  and  reassessment;  and  communication,  documentation,  and  coordination. 

The  risk  management  process  includes  the  following  key  activities,  performed  on  a  continuous  basis: 

a.  Risk  Identification 

b.  Risk  Analysis 

c.  Risk  Mitigation  Planning 

d.  Risk  Mitigation  Plan  Implementation 

e.  Risk  Tracking 
Significant  Accomplishment 

A  significant  accomplishment  is  a  specified  step  or  result  that  indicates  a  level  of  progress  toward 
completing  an  event  and,  in  turn,  meeting  the  objectives  and  requirements  of  the  contract. 

Significant  Accomplishment  Criteria 

Significant  accomplishment  criteria  are  specific,  measurable  conditions  that  must  be  satisfactorily 
demonstrated  before  a  significant  accomplishment  listed  in  an  Integrated  Master  Plan  (IMP)  is  complete 
and  before  work  dependent  on  the  accomplishment  can  proceed. 

Software  Reviews 

Software  reviews  are  a  set  of  software  build  reviews  held  to  find  potential  problems,  such  as  incomplete 
or  inconsistent  products  or  software  products  that  would  result  in  a  system  that  would  not  satisfy  its 
requirements  or  the  needs  of  its  users. 

Software  reviews  include,  for  example: 

a.  Software  Requirements  and  Architecture  Review  (SAR) 

b.  Software  Build  Plan  Review  (SBPR) 

c.  Software  Build  Requirements  Review  (SBRR) 

d.  Software  Build  Design  Review  (SBDR) 

e.  Software  Build  Test  Readiness  Review  (SBTRR) 
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f.  Software  Build  Test  Exit  Review  (SBTER) 

These  reviews  are  repeated  for  each  build. 

For  incremental  software  reviews  (where  the  requirements  are  all  known  up  front),  the  SBRR  would  be 
brief,  focusing  on  the  current  understanding  of  the  software  requirements  for  the  upcoming  build  and 
verify  that  no  requirements  need  to  change  for  the  upcoming  or  future  builds. 

Space  System 

Space  system  is  the  organization  and  integration  of  a  mix  of  ground  (mobile  or  stationary),  space 
(satellites  with  a  mix  of  payloads),  and  airborne  capabilities  or  systems  into  a  system  of  systems  with  an 
operational  and  functions  capability  that  is  greater  than  the  capabilities  of  its  constituent  parts. 

Spiral  Development 

Spiral  development  is  a  process  and  a  life  cycle  development  model  where  a  desired  capability  is 
identified,  but  the  end-state  requirements  are  not  known  at  program  initiation.  Those  requirements  are 
refined  through  demonstration  and  risk  management,  where  there  is  continuous  user  feedback  and  each 
increment  provides  the  user  the  best  possible  capability.  The  requirements  for  future  increments  depend 
on  feedback  from  users  and  technology  maturation.  Boehm1  defines  the  spiral  model  as  a  “Risk-driven 
process  model  generator.”  It  is  used  to  guide  multi-stakeholder  concurrent  engineering  of  software 
intensive  systems.  It  has  two  main  distinguishing  features.  One  is  a  cyclic  approach  for  incrementally 
growing  a  system’s  degree  of  definition  and  implementation  while  decreasing  its  degree  of  risk.  The  other 
is  a  set  of  anchor  point  milestones  for  ensuring  stakeholder  commitment  to  feasible  and  mutually 
satisfactory  system  solutions.” 

Synthesis 

Synthesis  is  the  process  whereby  the  functional  architectures  and  their  associated  requirements  are 
translated  into  physical  architectures  and  one  or  more  physical  sets  of  hardware,  software,  and  personnel 
solutions. 

System 

The  system  can  consist  of  a  System  of  Systems  (SoS),  Family  of  Systems  (FoS),  Segments,  Subsystems, 
and/or  Component. 

All  systems  consist  of  two  elements: 

a.  The  end  products,  to  be  used  by  an  acquirer  for  an  intended  purpose  and  the  set  of  enabling 
products  that  enable  the  creation,  realization,  and  use  of  an  end  product,  or  an  aggregation 
of  end  products 

b.  The  enabling  products,  used  to  perform  the  associated  process  functions  of  the  system — 
develop,  produce,  test,  deploy,  and  support  the  end  products;  train  the  operators  and 
maintenance  staff  is  using  the  end  products;  and  retire  or  dispose  of  end  products  that  are 
no  longer  viable  for  use 

Both  the  end  products  and  the  enabling  products  are  either  developed  or  reused,  as  appropriate.  The 
system  implicitly  includes  the  personnel  who  develop,  produce,  test,  operate,  support,  and  retire  the 
system  products,  as  well  as  both  those  who  train  others  involved  with  these  system  functions,  and  the 
human  factors  issues  and  concerns  associated  with  these  personnel.  Such  personnel  and  human  factors 
issues  are  included  in  the  applications  of  the  technical  review  processes  of  this  standard. 


1  Boehm,  Barry.  2001.  Understanding  the  Spiral  Model  as  a  Tool  for  Evolutionary  Acquisition. 
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Systems  Engineering  (SE) 

Systems  engineering  is  the  overarching  process  that  a  program  team  applies  to  transition  from  a  stated 
capability  need  to  an  operationally  effective  and  suitable  system. 

Systems  engineering  encompasses  the  application  of  systems  engineering  processes  across  the  acquisition 
life  cycle  (adapted  to  each  and  every  phase)  and  is  intended  to  be  the  integrating  mechanism  for  balanced 
solutions  addressing  capability  needs  and  design  considerations  and  constraints,  as  well  as  limitations 
imposed  by  technology,  budget,  and  schedule. 

Systems  Engineering  Processes 

Systems  engineering  processes  are  iterative  and  recursive  and  are  applied  at  lower  and  lower  levels  of  the 
system  structure  to  allow  an  orderly  progression  from  one  level  of  development  to  the  next  more  detailed 
level  through  the  use  of  controlled  baselines. 

a.  The  SE  processes  are  applied  early  in  concept  definition,  and  then  continuously  throughout 
the  total  life  cycle 

b.  The  SE  processes  are  used  for  the  system,  subsystems,  and  system  components  as  well  as 
for  the  supporting  or  enabling  systems  used  for  the  production,  operation,  training,  support, 
and  disposal  of  that  system 

The  SE  processes  enable  the  transition  of  requirements  from  design  to  system,  and  serve  as  an  integrated 
framework  within  which  the  universe  of  requirements  can  be,  as  a  collective  whole,  defined,  analyzed, 
decomposed,  traded,  managed,  allocated,  designed,  integrated,  tested,  fielded,  and  sustained. 

System  of  Systems  (SoS)  Engineering 

System  of  Systems  engineering  is  a  disciplined  process  that  encompasses  the  planning,  analysis, 
organization,  and  integration  of  a  mix  of  existing  and  new  capabilities/systems  into  a  system  of  systems 
capability  greater  than  the  sum  of  the  capabilities  of  the  constituent  parts. 

SoS  engineering  is  a  top-down,  comprehensive,  collaborative,  multidisciplinary,  iterative,  and  concurrent 
technical  management  process  that  is  used  for  identifying  system  of  systems  capabilities;  allocating  such 
capabilities  to  a  set  of  interdependent  systems;  and  coordinating  and  integrating  all  the  necessary 
development,  production,  sustainment,  and  other  activities  throughout  the  life  cycle  of  a  system  of 
systems. 

System  Verification  Review  (SVR) 

The  SVR  is  a  multidisciplinary  technical  review  conducted  to  ensure  that  the  system  under  review  can 
proceed  into  low-rate  initial  production  (LRIP)  and  full-rate  production  (FRP)  within  cost  (program 
budget),  schedule  (program  schedule),  risk,  and  other  system  constraints. 

The  SVR  confirms  that: 

a.  The  system  final  product,  as  evidenced  by  its  production  configuration,  meets  the 
functional  requirements  (derived  from  the  Capability  Development  Document  and  draft 
Capability  Production  Document)  as  documented  in  the  Functional,  Allocated,  and  Product 
Baselines 

b.  The  decision  database  has  been  maintained  to  capture  all  changes  and  updates  so  that  it 
completely  and  accurately  captures: 

(1)  The  current  approved  baselines 

(2)  Deficiencies  discovered  during  verification  (DT&E)  and  validation  (IOT&E)  have  been 
resolved  and  changes  approved  and  implemented 

(3)  All  other  approved  changes  have  been  incoiporated  into  the  affected  baselines  and  the 
affected  system  products  verified  to  comply 
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(4)  The  life  cycle  cost  projections  remain  consistent  with  the  program  affordability 
constraints 

(5)  The  requisite  plans,  procedures,  resources,  and  facilities  are  available  (or  on  schedule) 
to  initiate  production,  production  verification,  training,  deployment,  operations, 
support,  and  disposal 

(6)  The  remaining  risks  have  been  identified  and  can  be  handled  in  the  context  of  the 
planned  program 

(7)  The  verification  data  documents  the  system  and  its  constituent  products’  compliance 
with  the  baselines  and  satisfies  the  requirements,  the  allocated  baselines,  and  the  design 
release  baselines,  including  an  assessment  of  the  assumptions  and  methods  used  in 
verification  by  analysis 

SVR  is  often  conducted  concurrently  with  the  Production  Readiness  Review. 

SVR  provides  an  audit  trail  from  the  Critical  Design  Review. 

SVR  provides  inputs  to  the  Capability  Production  Document. 

Tailoring 

Tailoring  is  the  modification  of  text,  figures,  graphs,  or  tables  of  specifications,  standards,  and  other 
requirements  or  tasking  documents  to  clarify  the  extent  to  which  they  are  applicable  to  a  specific 
acquisition  contract. 

a.  Tailoring  is  applied  at  the  discretion  of  the  contracting  agent.  In  each  application,  this 
document  shall  be  tailored  to  the  specific  requirements  of  a  particular  program,  program 
phase,  or  contractual  structure  as  directed  by  the  contracting  agent.  Tasks  that  add 
unnecessary  costs,  data,  and  any  factors  that  do  not  add  value  to  the  process  or  product 
should  be  eliminated 

b.  Tailoring  takes  the  form  of  deletion  (removal  of  tasks  not  applicable),  alteration  (modifying 
tasks  to  more  explicitly  reflect  the  application  to  a  particular  effort),  or  addition  (adding 
tasks  to  satisfy  program  requirements) 

Tailoring  of  requirements  and  task  statements  may  be  used  in  preparing  solicitation  documents  as  well  as 
by  contractors  in  response  to  a  draft  Request  for  Proposal. 

TBD/TBR/TBS/TBP 

a.  TBD  -  To  be  determined  by  the  developer  (or  formally  recommended  to  the  contracting 
agent)  based  on  analysis  or  test  by  a  stated  and  documented  date 

b.  TBR  -  The  preliminary  element  shall  be  resolved  by  the  developer  (or  recommended  to  the 
contracting  agent)  based  on  analysis  or  test  by  a  stated  and  documented  date 

c.  TBS  -  To  be  supplied  by  the  contracting  agent  to  the  developer  by  an  agreed-to  and 
documented  date 

d.  TBP  -  To  be  provided  by  the  contracting  agent  to  the  developer  by  an  agreed-to  and 
documented  date 

Technical  Performance  Measure  (TPM) 

TPM  is  a  measurement  comparing  the  current  actual  achievement  for  technical  parameters  with  that 
anticipated  at  the  current  time  and  on  future  dates  and  confirms  progress  and  identifies  deficiencies  that 
may  jeopardize  meeting  a  requirement  or  delivery  of  a  KPP  capability. 

Typical  TPM  candidate  selections: 

a.  Performance  parameters  that  significantly  qualify  the  entire  system 

b.  Parameters  directly  derived  from  analyses,  demonstrations,  or  test 
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c.  A  direct  measure  of  value  derived  from  results  of  analyses  or  tests 

d.  Predicted  values  that  have  a  basis  (analyses,  historical  data) 

e.  Each  parameter  can  periodically  be  measured  and  profiled  to  compare  with  predicted  values 
and  tolerances  over  the  program  life  cycle 

Tracking  of  TPMs  typically  begins  as  soon  as  a  baseline  design  has  been  established,  which  ideally 
occurs  by  SFR  but  no  later  than  PDR  even  though  the  full  set  of  selected  TPMs  may  not  be  available  until 
later  in  the  program  life  cycle 

Technology  Readiness  Levels  (TRLs) 

TRL  is  a  measure  of  the  evolving  technologies  (materials,  components,  devices,  etc.)  maturity  prior  to 
incorporating  that  technology  into  an  end  item  (system,  subsystem,  or  Cl). 

Table  6.  DoD  TRL  Definitions 


TRL 

Definitions 

Level  1 

Basic  principles  observed  and  reported 

Level  2 

Technology  concept  and/or  application  formulated 

Level  3 

Analytical  and  experimental  critical  function  and/or  characteristic  proof  of  concept 

Level  4 

Component  and/or  breadboard  validation  in  laboratory  environment 

Level  5 

Component  and/or  breadboard  validation  in  relevant  environment 

Level  6 

Sy stem/ subsystem  model  or  prototype  demonstration  in  a  relevant  environment 
(Ground  or  Space) 

Level  7 

System  prototype  demonstration  in  an  operational  (space)  environment 

Level  8 

Actual  system  completed  and  (flight)  qualified  through  test  and  demonstration 
(Ground  and  Space) 

Level  9 

Actual  system  (flight)  proven  through  successful  mission  operations 
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Figure  7.  TRL  Relationship  to  Systems  Development  Maturity 
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Table  7.  DoD  Hardware  and  Software  TRL  Descriptions 


TRL 

Hardware  and  Software  Descriptions 

Level  1 

Lowest  level  of  technology  readiness.  Research  begins  to  be  translated  into  applied  research 
and  development.  Examples  might  include  paper  studies  of  a  technology’s  basic  properties. 

Level  2 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be  invented. 
Applications  are  speculative,  and  there  may  be  no  proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic  studies. 

Level  3 

Active  research  and  development  is  initiated.  This  includes  analytical  studies  and  laboratory 
studies  to  physically  validate  analytical  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or  representative. 

Level  4 

Basic  technological  components  are  integrated  to  establish  that  they  will  work  together.  This 
is  relatively  “low  fidelity”  compared  to  the  eventual  system.  Examples  include  integration 
of  “ad  hoc”  hardware  in  the  laboratory. 

Level  5 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  it  can  be  tested 
in  a  simulated  environment.  Examples  include  “high-fidelity”  laboratory  integration  of 
components. 

Level  6 

Representative  model  or  prototype  system,  which  is  well  beyond  that  of  TRL  5,  is  tested  in 
a  relevant  environment.  Represents  a  major  step  up  in  a  technology’s  demonstrated 
readiness.  Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory  environment  or 
in  simulated  operational  environment. 

Level  7 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from  TRL  6, 
requiring  demonstration  of  an  actual  system  prototype  in  an  operational  environment,  such 
as  in  aircraft,  vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

Level  8 

Technology  proven  to  work  in  its  final  form  and  under  expected  conditions.  In  most  cases, 
this  TRL  represents  the  end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended  weapon  system  to  determine  if  it  meets 
specifications. 

Level  9 

Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions,  such  as 
those  encountered  in  operational  test  and  evaluation.  Examples  include  using  the  system 
under  operational  mission  conditions. 

Traceability 

Traceability  is  the  ability  to  relate  an  element  of  the  requirements  baseline,  functional  architecture, 
allocated  baseline,  design  release  baseline,  and  product  configuration  baseline  or  their  representation  in 
the  decision  database  and  their  relationship  to  any  other  element.  Traceability  is  inherently  bidirectional: 

Downward  Traceability:  where  a  master-subordinate  or  parent-child  relationship  exists 

Upward  Traceability:  where  a  subordinate-master  or  child-parent  relationship  exists. 

Validation 

Validation  is  the  confirmation,  through  the  provision  of  objective  evidence  that  the  requirements  for  a 
specific  intended  use  or  application  have  been  fulfilled  DoDI  5000.02,  e.g.: 

a.  The  demonstration  that  the  end  item  has  its  required  attributes,  that  any  assumptions 
necessary  in  its  development  are  valid  (i.e.,  acceptable  to  the  customer),  and  that  the 
effectiveness  of  the  emerging  system  design  can  affordably  satisfy  the  system  technical 
requirements  and  constraint 
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b.  The  process  of  demonstrating  that  the  El  or  product  component  fulfills  its  intended  use 
when  placed  in  its  target  environment 

Verification 

Verification  is  the  confirmation,  through  the  provision  of  objective  evidence  that  specified  requirements 
have  been  fulfilled  [ISO  9000:  2000].  Verification  of  a  product  demonstrates  that  the  product  has  been 
built  to  the  validated  design  and  performance  baseline  and  meets  the  product’s  specified  requirements.  It 
is  the  process  that  confirms  that  the  El  meets  the  design-to  or  build-to  specification. 

Waterfall  Development 

Waterfall  development  is  a  process  and  life  cycle  model  in  which  all  of  the  requirements  are  known  at  the 
beginning,  with  the  requirements,  architecture,  design,  implementation,  integration,  and  test  activities 
performed  in  that  order,  possibly  with  feedback  and  overlap. 
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a.k.a. 

also  known  as 

ABL 

Allocated  Baseline 

ACAT 

Acquisition  Category 

AFOTEC 

AF  Operations  Test  and  Evaluation  Center 

AI 

Action  Item 

AoA 

Analysis  of  Alternatives 

APB 

Acquisition  Program  Baseline 

ASR 

Alternative  Systems  Review 

AT 

Acquisition  Team 

B/L 

Baseline 

BDP 

Burn  Down  Plan 

BL 

Baseline 

BOL 

Beginning  of  Life 

C4I 

Command,  Control,  Communications,  Computers,  and  Intelligence 

CAIV 

Cost  As  an  Independent  Variable 

CAO 

Contract  Administration  Office 

CARD 

Cost  Analysis  Requirements  Description 

CDD 

Capability  Development  Document 

CDM 

Contractor  Data  Manual 

CDR 

Critical  Design  Review 

CDRL 

Contractor  Data  Requirement  List 

CDS 

Contractor  Data  Sheet 

Cl 

Configuration  Item 

CM 

Configuration  Management 

CMMI 

Capability  Maturity  Model  Integration 

COMSEC 

Communications  Security 

CONOPS 

Concept  of  Operation 

COTS 

Commercial  off  the  Shelf 

CRD 

Capstone  Requirements  Document 

CSCI 

Computer  Software  Critical  Item 

CSDM 

Computer  System  Diagnostic  Manual 

CSOM 

Computer  System  Operator’s  Manual 

CTE 

Critical  Technology  Element 

CWBS 

Contract  Work  Breakdown  Structure 

DAR 

Defense  Acquisition  Regulation 

DAS 

Direct  Attached  Storage 
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DDB 

Decision  Data  Base 

DIA 

Defense  Intelligence  Agency 

DIACAP 

Defense  Intelligence  Assessment  Capability 

DID 

Data  Item  Description 

DISR 

DoD  Information  Standards  Repository 

DMS 

Diminishing  Manufacturing  Source 

DMSMS 

Diminishing  Manufacturing  Sources  and  Material  Shortages 

DoDAF 

Department  of  Defense  Architecture  Framework 

DT&E 

Development  Test  and  Evaluation 

DUSD 

Deputy  Under  Secretary  of  Defense 

ECO 

Engineering  Change  Order 

ED 

Engineering  Development 

EDMO 

Engineering  Data  Management  Officer 

EESS 

Environmental  Effects  Stress  Screening 

El 

End  Item 

EMC 

Electromagnetic  Compatibility 

EMI 

Electromagnetic  Interference 

EOL 

End  of  life 

EPDS 

Electrical  Power  Distribution  System 

ES&OH 

Environmental  Safety  and  Occupational  Health 

ESD 

Electrostatic  Discharge  Control 

ESLOC 

Equivalent  Source  Lines  of  Code 

EVM 

Earned  Value  Management 

F/A 

Fabrication/Assembly 

FAR 

Federal  Acquisition  Regulation 

FBL 

Functional  Baseline 

FCA 

Functional  Configuration  Audit 

FF&F 

Form,  Fit  and  Function 

FFBD 

Functional  Flow  Block  Diagram 

FFF&P 

Form,  Fit  (suitability),  Function,  and  Performance 

FFFP 

Form,  Fit,  Function,  Performance 

FMECA 

Failure  Modes,  Effects,  and  Criticality  Analysis 

FoS 

Family  of  Systems 

FQR 

Formal  Qualification  Review 

FRP 

Full-Rate  Production 

FSM 

Firmware  Support  Manual 

GFE 

Government-Furnished  Equipment 

GIG 

Global  Information  Grid 
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GOTS 

Government  off  the  Shelf 

GSE 

Ground  Support  Equipment 

GUI 

Graphical  User  Interface 

HAL 

Hardware  Allocation  Listing 

HAR 

Hardware  Acceptance  Review 

HCI 

Human  Computer  Interface 

HMMP 

Hazardous  Materials  Management  Plan 

HSI 

Human  Systems  Integration 

HW 

Hardware 

HWCI 

Hardware  Configuration  Item 

I&T 

Installations  and  Test 

I/F 

Interface 

IA 

Information  Assurance 

IAW 

In  Accordance  With 

IB 

Industrial  Base 

IBR 

Integrated  Baseline  Review 

ICD 

Initial  Capabilities  Document 

IMP 

Integrated  Master  Plan 

IMS 

Integrated  Master  Schedule 

IOT&E 

Initial  Operational  Test  and  Evaluation 

IPT 

Integrated  Product  Team 

IRS 

Interface  Requirements  Specification 

ISO 

International  Organization  for  Standardization 

ISP 

Integrated  Support  Plan 

ISR 

In-Service  Review 

IT 

Information  Technology 

ITR 

Initial  Technical  Review 

IV&V 

Independent  Verification  and  Validation 

KDP 

Key  Decision  Point 

KIP 

Key  Interface  Profile 

KPP 

Key  Performance  Parameter 

LCC 

Life  Cycle  Cost 

LMI 

Logistics  Management  Information 

LRIP 

Low-Rate  Initial  Production 

M&S 

Modeling  and  Simulation 

M/F/LRR 

Mission,  Flight,  and/or  Launch  Readiness  Reviews 

M/PRR 

Manufacturing  and  Product  Readiness  Review 

MDA 

Milestone  Decision  Authority 
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Military 

MLS 

Multilevel  Secure 

MRL 

Manufacturing  Readiness  Levels 

MRL 

Material  Requirements  List 

MRR 

Manufacturing  Readiness  Review 

MTBF 

Mean  Time  Between  Failure 

MTTR 

Mean  Time  to  Repair 

NAS 

Network  Attached  Storage 

NCOW 

Net-Centric  Operations  and  Warfare 

NCOW-R 

Net-Centric  Operations  and  Warfare-Reference 

NDI 

Nondevelopmental  Item 

NEPA 

National  Environmental  Policy  Act 

NR-KPP 

Net-Ready  Key  Performance  Parameter 

NSS 

National  Security  Strategy 

NTIB 

National  Technology  and  Industrial  Base 

OA 

Operational  Architecture 

OCD 

Operational  Capability  Demonstration 

OpsCon 

Operations  Concept 

OSHA 

Occupational  Safety  and  Health  Act/Administration 

OT&E 

Operational  Test  and  Evaluation 

OTRR 

Operational  Test  Readiness  Review 

OTS 

Off-the-Shelf 

OV 

Operational  View 

PCA 

Physical  Configuration  Audit 

PCO 

Procuring  Contracting  Officer 

PCR 

Physical  Configuration  Review 

PDR 

Preliminary  Design  Review 

PEL 

Permissible  Exposure  Level 

PESHE 

Programmatic  Environmental,  Safety,  and  Occupational  Health  Evaluation 

PHS&T 

Packaging,  Handling,  Storage  and  Transportability 

PM&P 

Parts,  Materials,  and  Processes 

POC 

Point  of  Contact 

PP 

Program  Protection 

PPP 

Program  Protection  Plan 

PRR 

Production  Readiness  Review 

PSP 

Product  Support  Plan 

PWBS 

Program  Work  Breakdown  Structure 

R&M 

Reliability  and  Maintainability 
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RAD 

Requirements  Allocation  Document 

RAID 

Redundant  Array  of  Inexpensive  Disk 

RAM&T 

Reliability,  Availability,  Maintainability,  and  Testability 

RBL 

Requirements  Baseline 

RD&M 

Reliability,  Dependability,  and  Maintainability 

RFI 

Radio  Frequency  Interface 

RM 

Risk  Management 

RM&M 

Risk  Management  and  Mitigation 

RRDD 

Risk  Reduction  and  Design  Development 

RTM 

Requirements  Traceability  Matrix 

RVTM 

Requirements  Verification  Traceability  Matrix 

S&T 

Science  and  Technology 

SAN 

Security  Assistance  Network 

SAR 

Software  Requirement  and  Architecture  Review 

SCAMPI 

Standard  CMMI  Appraisal  Method  for  Process  Improvement 

SDD 

System  Development  and  Demonstration 

SDP 

Software  Development  Plan 

SEMP 

System  Engineering  Management  Plan 

SEP 

Systems  Engineering  Plan 

SFR 

System  Functional  Review 

SGLS 

Space-to-Ground  Link  Set 

SGLS 

Space-to-Ground  Link  System 

SLOC 

Source  Lines  of  Code 

SMC 

Space  and  Missile  Systems  Center 

SME 

Subject  Matter  Expert 

SoS 

System  of  Systems 

SOW 

Statement  of  Work 

SPM 

Computer  Programmer’s  Manual 

SPO 

System  Program  Office 

SRDR-F 

Final  Developer  Report  and  Data  Dictionary 

SRDR-I 

Initial  Developer  Report  and  Data  Dictionary 

SRR 

System  Requirements  Review 

SRS 

Software  Requirements  Specification 

SSA 

Storage  System  Architecture 

SSE 

System  Security  Engineering 

STD 

Standard 

SUM 

Software  User’s  Manual 

sv 

System  View 
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SVR 

System  Verification  Review 

SW 

Software 

SWCI 

Software  Configuration  Item 

T&E 

Test  and  Evaluation 

TBD 

To  Be  Determined 

TBR 

To  Be  Resolved 

TBS 

To  Be  Supplied 

TD 

Technology  Development  and/or  Technology  Demonstration 

TDP 

Technology  Development  Phase 

TDP 

Technical  Data  Package 

TEMP 

Test  and  Evaluation  Master  Plan 

TOR 

Technical  Operating  Report 

TPM 

Technical  Performance  Measurement 

TRA 

Technology  Readiness  Assessment 

TRC 

Technical  Review  Criteria 

TRD 

Technical  Requirements  Document 

TRL 

Technology  Readiness  Level 

TRR 

Test  Readiness  Review 

TV 

Technical  View 

USAF 

United  States  Air  Force 

v&y 

Verification  and  Validation 

VCRM 

Verification  Cross  Reference  Matrix 

WBS 

Work  Breakdown  Structure 
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SMC  Standard  Improvement  Proposal 

INSTRUCTIONS 

1.  Complete  blocks  1  through  7.  All  blocks  must  be  completed. 

2.  Send  to  the  Preparing  Activity  specified  in  block  8. 

NOTE:  Do  not  be  used  to  request  copies  of  documents,  or  to  request  waivers,  or  clarification  of  requirements  on 
current  contracts.  Comments  submitted  on  this  form  do  not  constitute  or  imply  authorization  to  waive  any  portion  of 
the  referenced  document(s)  or  to  amend  contractual  requirements.  Comments  submitted  on  this  form  do  not 
constitute  a  commitment  by  the  Preparing  Activity  to  implement  the  suggestion;  the  Preparing  Authority  will 
coordinate  a  review  of  the  comment  and  provide  disposition  to  the  comment  submitter  specified  in  Block  6. 
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